Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes new metrics based on the activation patterns of a model. It propose an activation pattern entropy metric and a graph theoretic metric of neuron communities. The authors then evaluated models trained on MNIST, Fashion MNIST, Fashion MNIST mixed, CIFAR-10, and plant village using their new metrics. They found that entropy is negatively correlated with training accuracy, modularity is correlated with training accuracy, and that the number of well defined neural communities increase with accuracy.
main_review: Strengths:
- The metrics and evaluation procedures are clearly explained. Developing new metrics to measure models activations and relating them to model performance an exciting research direction. 

Weaknesses:
- The metrics introduced in this paper do not immediately suggest a way to build higher performance models or provide significant insight into what the model's have learned. Just because a representational metric is correlated with a model's performance does not mean that it causes a model's performance improvement. It also does not mean that a representational metric is predictive of the performance of the fully trained model. I would be more convinced that these are important metrics if they could be incorporated as a regularization term in a loss function (Ex. Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs (Leavitt & Morcos 2020)) or they could be used to predict which models will generalize better (Ex. Fantastic Generalization Measures and Where to Find Them (Jiang et al. 2019))
- I do not think that the entropy metric is entirely novel. Many past studies have demonstrated that individual units in neural networks become specialized during the course of training for detecting specific features or classes. (Ex. Understanding the Role of Individual Units in a Deep Network (Bau et. al. 2020))
- The finding that modularity is correlated with training accuracy, and that the number of well defined neural communities increase with accuracy could only apply to a vanilla SGD model training setup. Model representations are often very flexible and it could be that regularization against modularity results in a model that is still equally high performance, but is much less modular. Such an experiment could demonstrate if the metrics studied in this paper are measuring an important model representation structure necessary for high performance or a representational coincidence of their particular training method.


summary_of_the_review: Overall, the metrics in this paper are sensible and clearly explained, but further experiments are necessary to demonstrate insights into what models are learning or what directions for performance improvements the metrics can suggest.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper examined novel neuron activation pattern analysis on neural network classification models via graph theoretic and entropy-based methods. The authors showed reliability of their work by approving hypotheses on examining the qualitative correlation between model performance and activation patterns. The main technical contribution of this paper comes from explaining the neural classifiers by combining the graph-theoretic and information-theoretic approaches.
main_review: Strengths
- Suitable and persuasive application of the graph and community structure to explain  the activation behavior and relations of the neurons.
- Appropriate usage of the entropy and modularity score to effectively measure the tendency and the predictability of the neuron activation.
- Intuitive hypotheses to validate the correlation between suggested methods and the behavior of the neurons.
Weaknesses
- As authors mentioned in the discussion section(8: Limitations and Future Work). The scalability of this work is suspicious. There ‘s no guarantee that proposed activation analysis methods will also show the same correlation tendency in other neural networks with more complicated architectures than  multi-layer perceptron networks.
- To compute the entropy, activation pattern matrix is required for each layer. The spatial complexity of the matrix is the multiplication of the total number of neurons and the size of a training data. Computational overload will be inevitable when it’s trained on larger data and models.
- Aside from the architecture, the authors only had done experiments on the dropout conditions. Experiments on other various hyperparameters/conditions that can affect neuron configurations or learning ability of the neural network(e.g., pruning) could add more validities to their ‘comprehensive experimental study’.

summary_of_the_review: The suggested explanatory methodologies and approaches were pretty persuasive to explain the network behavior in microscopic context. Nevertheless, the scalability and practicality of the work should be examined carefully.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation
patterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.
main_review: The authors propose two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation
patterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.

The paper is interesting, easy to read, and very well-structured. Furthermore, the subject of the paper is very relevant: the exploration of tools to increase the understanding and explainability of neural models. However, I have several important questions/comments to the authors:

1) In page 6, the authors clearly state that "we only used fully connected (FC) and dropout layers". However, in the supplementary material (page 1), the authors present a graphical representation of the deep learning model architecture used, and they include convolutional blocks. Three questions in this regard:
- do the authors employ or do not employ only fully connected layers?
- if what is stated in the paper is correct, why to employ only fully connected layers if the problem tackled is image classification? I think it would be more natural to employ ConvNets. 
- do the authors think that the conclusions extracted would also apply to ConvNets and/or other neural models and problems?

2) The authors discuss three main hypotheses:
H1: The neurons that are frequently activated together form a community in the activation pattern graph.
H2: The modularity of the activation pattern graph is related to a deep learning model's performance.
H3: The entropy of the activation pattern is related to a deep learning model's performance.

Regarding H1, the authors conclude that "for models with good performance, the number of well defined communities increases with training". In turn, H2 can contribute to find representative neuron sets for different classes, and authors conclude that "the communities' quality is positively correlated with training accuracy". However, if I'm not mistaken, the main conclusion related to H3 is that, at the beginning of the training, due to the randomness of the initial weights, the activation of the neurons becomes unpredictable (high entropy); while, as the training moves forward, the activation pattern is biased and thus the entropy will decrease. It is not totally clear to me how this hypothesis (H3), and the associated experimentation and conclusions, effectively contribute to a better understanding and evaluation of the performance and explainability of neural networks. 

3) Figure 3 of the supplementary material shows that the entropy, first, decreases with iterations (which fits the hypothesis managed in the paper: "As the training progresses, the entropy should decrease, representing a biased neuron activation behavior") but from the 8th-9th iteration it dramatically increases. What is the explanation of the authors for this, and how this fits the initial assumption?

4) All experimental evaluation is performed on quite "small" architectures. Is it possible that the explanatory and exploratory capacities of the presented methods are diminished with much larger/deeper models? 

5) Also regarding the experimental configuration, it is unclear what exactly an "iteration" is, or what are the different hyperparameters involved (learning rate, batch size, etc. and how these were selected). In other words, I have the feeling that the results are hardly reproducible. On the other hand, there are no competitor approaches to compare the performance obtained by these metrics/methods. It would be interesting to see how the techniques presented in this paper compare to other techniques already present in the literature. 


summary_of_the_review: Interesting paper and relevant subject. But I think there are some inconsistencies in the paper, and the overall impact, conclusions and applicability of the paper is somewhat dubious.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: An important problem of understanding the performance of deep learning models is studied by this submission. Two graph-theoretic and information-theoretic metrics are checked in some datasets with MLP networks. Preliminary results with expected correlation are observed.
main_review: Understanding why deep learning works by inspecting the internal representations beyond loss value or performance (accuracy) is an important topic. This paper tries to advance our understanding with two newly proposed metrics, which is a good point. The writting and logic is clear.

However, several major concerns are obvious:

1. Not enough literature survey. The authors claim this paper "lay the initial groundwork for graph and entropy-based studies to analyze the deep learning models’ performance", but entropy is prevalent in deep learning. Given the bold literature on both understanding deep learning and entropy utilization, it is hard to say this paper is the first to do so.

2. No baseline methods to compare the significance of the two proposed metrics. There are many existing approaches in explaining deep learning models, such as turning deep models into decision trees in "Distilling a Neural Network Into a Soft Decision Tree". Then complexity measurement of the decision tree can be a baseline. Or counting the number of linear regions induced by neural networks like "On the number of linear regions of deep neural networks" is also a feasible idea. Frankly speaking, I'm not an expert in explaining deep learning, but I believe there have exist many methods to explain deep models' performance. The significance of methods proposed in this paper should be compared against existing approaches.

3. Experiments are rather preliminary, which only involves some toy datasets and MLP networks. It is not clear whether the method scales to larger and real-world problems, as admitted by authors in the paper.

By the way, I'm curious about how the experiments are done. In section 6, what do you mean by "iteration"? Is it one mini-batch or one epoch? What is the batch-size? The results are quite strange because I see that 99.56% accuracy is achieved within just 20 iterations.
summary_of_the_review: I acknowledge the importance of the research direction in this paper. The contribution of this paper is not properly compared against existing methods, making it difficult to tell their significance. Meanwhile, the toy setting in experiments makes me doubt whether the techniques can be extended to real-world scenarios.
review_title: null

Meta-review: Understanding neural networks once they have been trained is a big open problem for machine learning. This manuscript designed graph theoretic and information theoretic measures aimed at helping us understand community structure and function in trained networks. In particular, they measure community structure (modularity) and entropy for trained networks and related these to the performance of the networks. The manuscript runs experiments with fully connected networks on problems such as MNIST and CIFAR. Both community structure and entropy measures are shown to correlate (Spearman and Pearson correlation coefficients) with performance metrics in the networks studied. 
Reviewers tended to agree that the paper was well written and motivated by an interesting and timely question (understanding trained networks). However, on the whole, most of the reviewers believe that the manuscript is too preliminary for publication at ICLR and I agree. A central issue cited by most of the reviewers is that the experiments are performed on small/toy models for small tasks and under particular hyperparameter regimes. It is therefore unclear to what extent the results would generalize to other situations. E.g. would the results hold for larger dataset or for convolutional neural networks? Connected to this complaint, reviewers worry that there is not enough connection to the literature and baseline methods that could be used to predict performance given measures of trained network activity. Even allowing that the observed correlations are true and generalizable, are these measures better than those covered elsewhere in the literature? Additionally problematic, the measures are not theoretically justified either. Thus, we are missing both reasoned arguments for the metrics and robust quantification beyond a limitted experimental setting. One reviewer, Xmnm, is compelled by the work and recommends acceptance. However, they do not present a compelling case for acceptance, and even repeat several of the concerns raised by other reviewers. 
In sum, the work is on an interesting subject and timely, but needs further work to be ready for publication.
Decision: Reject