Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to analyse DNNs' generalization behaviours by looking at activation patterns, in particular the entropy of a neuron, and the mutual information between neurons. Experiments show that these measures rank different models similarly to other extrinsic measures (which require labels). This is shown by perturbing labels and training data distributions, leading to two extremes (1) spurious correlations and (2) example memorization.

An interesting finding of this paper is that a "normal" neural network's entropy and MI seems to be in a sweet spot. Too much MI and the network is probably having spurious correlations, too little, and it's probably memorizing individual examples.
strengths_and_weaknesses: Originality: This paper is not the first to analyse entropies and mutual information within neural networks. The specific instantiation and what's being tested are novel, and hopefully informative to the community, but not wholly surprising.

Quality/Clarity: The experiments in the paper are fairly well explained and executed (although I have some concerns), and the thesis is very clear.

Significance: These findings are certainly interesting, but I'm not sure they are groundbreaking either. The authors suggest that the proposed measures can be used to rank models and thus perform model selection, but this result feels a bit forced, since the authors appear to compare models which they specifically train with various algorithmic changes/perturbations that might cause this.

questions: Concerns:

I'm not sure I understand what entropy is being computed, or if it even is an entropy. Shouldn't you want to compute the entropy of the empirical categorical distribution of $A_i$ over the dataset, when $A_i$ is discretized into bins? The sum in the entropy calculation would then be over the number of bins, not the number of examples $S$. $H(A_i)$ should be
$$H(A_i) = \sum_{j=1}^{N_{bins}} p(\hat{a} _ {i}^j) \log p(\hat{a} _ {i}^j)$$
where 

$$p(\hat{a}^j_i) = \frac{1}{S}\sum_k^S 1[\mathrm{bin}(a_{(i,k)}) ==  j]$$

Would it be possible to reproduce the plots in the paper with the above computation? -- I'd really like to understand this before accepting this paper.

> [L98] we fit a Gaussian mixture model over all values

Why not just plot the histogram of values? Seems like this could hide some artefacts.

As mentioned above, what models are Table 2's values computed on? Would a practitioner looking to apply this model selection see this by, say, just changing the learning rate or capacity of a model?

limitations: The authors address social impacts, but limitations are limited to future work rather than fundamental weaknesses of the proposed approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigate the detection of memorization. The author categorize the memorization into 2 kinds: heuristic based and example based. The main hypothesis is, compared with the base case, the neural activations of heuristic based memorization has lower entropy and low MI, while the other one has higher entropy and MI. This is empirically and qualitatively validated on both language and vision tasks, and the author provide a quantitative analysis of this metric as a way for model selection.
strengths_and_weaknesses: Strength:
The paper is easy to follow and well presented. The idea of observing neural activities for memorization is interesting, and I think proposition of MI and inner neurons entropy could be useful for researchers interested this topic. The experiments are solid to show the soundness of the hypothesis.

Weakness:
The main critique for me is that, while the model selection application is given, I am not sure how good that number is. There are some other baselines on ranking models based on generalization metric, e.g., [1], [2]. I would like to see more comparison with some baselines from these work to see the value of model selection.


[1] https://arxiv.org/abs/1912.02178
[2] https://arxiv.org/abs/2012.07976
questions: 1. Is entropy and MI strongly correlated? If so can we just use one of these metric?
2. Can you provide some insights on how this can be useful in designing a regularizer for training?
limitations: Not that I can see.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: 
This paper studied two types of memorization that impedes the generalization, the heuristic memorization and the example-level memorization, respectively.  This paper proposed that diversity in the activation pattern is linked to indicate these memorizations through information-theoretic measures.

To study the cause of the two types of memorizations, this paper establish the analysis from information theory side, it introduced two information theory measurements: the cross-sample diversity (i.e. entropy) and the cross-neuron diversity (MI). It then showed that heuristic memorization is linked to low cross-sample and cross-neuron diversity, while the example level memorization is linked to the both high value for these two diversities. The authors then demonstrated a potential application of this framework for model selection.

strengths_and_weaknesses: 
This paper is overall well-written in clarity. The attacked problem, model memorization quatification, is a theoretically important task. This paper proposed one way to understand the neural network behavior with grounded tools.


As one limitation, the author claimed in line 264 that this model could be used point-wise to characterize model behavior over individual examples, potentially useful for OOD detection; indeed there is a practical gap before pointwise application, as when people build a single model (drawn one point from the model architecture space) with arbitrary shape, it is easy to measure the entropy and MI but difficult to tell whether these number indicates a high memorization ratio. When people drawn multiple samples from model architecture space, it is also hard to tell whether these meaturements remains descriptive across heterogeneous architectures.

As the metric values are relative references not direct pointer, the authors are encoraged to use a subsection to discuss about the applicabilicable range and the possible issue in practical run.

On the other hand, observed from the results in Fig 2/3, the argued relationship can be outweighted than the variance except when the handcraft ratio alpha is extremely large. Similar observation can be found in Fig 6/7, where the pattenr of the mean of entropy and MI is observable, but the variance remains big. Given this, an independent subsection in the paper for more discussions and adequate claim is recommended.

questions: The applicability of these two metrics across heterogeneous architectures are under-discussed. Based on my understanding, the table 3 in the supple line 565 is only describing architectures within one model setting. If one trained two models with different architectures, could these metrics stil be comparable? 

Minors:

line 153:  "This is done" capital T.
limitations: There is no obvious negative societal impact given the theoretical nature of this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper studies two types of memorization: i) heuristic memorization and ii) example-level memorization. The authors study the diversity of activations units of the neural networks either across a set of examples (intra-neuron) or for the same example (inter-neuron). They observe, through various experiments on both synthetic and real datasets, that models experiencing heuristic memorization have lower diversity in their activations compared to models experiencing example-level memorization. They also suggest using their two metrics for model selection.
strengths_and_weaknesses: Strengths:

1- The paper is well-written and the figures look nice.

2- To the best of my knowledge, the proposed metrics to distinguish between two types of memorization are novel.

3- The empirical results include both vision and language tasks, and also both synthetic and real datasets.

Weaknesses:

1- The application for the method is not clear. It is also not clear what are the particular insights that this study brings to the table; is it that "models experiencing example-level memorization are more complicated than models experiencing heuristic memorization", and that "models experiencing heuristic memorization are less complicated than models not experiencing that"? Aren't these conclusions expected from our understandings of neural networks? What is the new outcome of this study? 

2- Comparison with other complexity measures is missing. If the activation diversity is simply capturing the complexity of the learned model, then how does it compare to other neural network complexity measures? What would be the specific advantage of these new metrics? 

3- Lack of comparison to other methods also applies to the model selection section. How does the model compare to other generalization measures? Also, the obtained Kendall tau is computed for which models? What is the scope of this model selection? If it's just computed over a few networks then the generalizability would be questionable.
questions: 1- In Figure 1, how would you compare the generalization ability of the three networks that are trained on the three tasks? Would you assume that the first model (trained on the base task) is the most "generalizable"? If so, then how would you explain Figure 1 (g) and (h) in terms of capturing generalization? Ranking the models based on these two figures gives us: model (b) > model (a) > model (c). Hence, if Neuron Entropy and Density of Neuron Pairs are reflective of generalization, this result would suggest that the model with "heuristic memorization" is the most "generalizable", which is counterintuitive.

2- The paper tries to distinguish heuristic memorization from example-level memorization. But what is not clear is how to distinguish memorization (either type) from no memorization? If the method is not able to do so, then I would suggest combining this approach with a method that detects memorization. That is, first, detect with that method whether the model is experiencing any memorization or not. Next, if the model is memorizing, then distinguish its type using your approach (low/high entropy/MI). This would be a much more interesting tool.

3- One concern raised in the introduction is that it's difficult to generate OOD datasets. How is this solved in your paper?

4- Why are the activation units of only the encoder considered? Why not use both the encoder and the decoder? What is even meant here by encoder and decoder? (lines 58-63)

5- How do you make sure that the models trained on synthetic datasets with heuristic memorization do actually memorize? This is important to double-check. Because it could be that the network chooses to classify based on the more complex features (for example the circles in Figure 1) rather than based on the simple induced feature. If that's the case, then studying that model is not the same as studying heuristic memorization. The validation accuracy shows this to some extent, but it would be better to see some examples as well. For example, for a new green point in the outer circle but high in the induced feature, what is the label?
limitations: The authors have not mentioned the limitations of their method in the main paper. But in the appendix, they mention potential societal impacts.
review_title: null

Meta-review: metareview: This paper analyzes the neural network representations using tools from information theory, and show that it could reflect memory patterns and distinguish between "heuristic memorization" and "example-level memorization". The reviewers generally find those findings novel and interesting. There were some concerns such as missing baseline comparison with other model selection criterions. The authors provided additional results in the rebuttals to address those concerns and improved the paper writing according to the reviewers' comments.
confidence: Less certain
Decision: Accept