Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors identify putative clusters of units/neurons in deep networks using spectral clustering on a graph defined by synaptic weights. The authors then argue that these structurally defined clusters of neurons have similar *functional representations*. Finding interpretable relationships between weight matrices and functional modules is challenging, and the authors should be applauded for attempting to tackle this challenging problem that few research groups are devoting energy to.

Despite these positive notes, I have reservations about the presentation and results of the paper. My main concerns are:

(1) The results are largely qualitative and anecdotal. In figure 1, for example, the authors show slightly higher contrast in their identified clusters than random clusters. The results are limited to black and white images (MNIST and fashion-MNIST), and not all examples look great. Only 2 figures are shown in the main paper, with a lot of other details shoved into the supplement. Thus, the writing and presentation could be improved to highlight the most exciting and surprising findings.

(2) The results crucially rely on a second paper which was concurrently submitted and can't be reviewed because it is anonymized. The results shown in this paper are thin and qualitative (see point 1), so in my view these two paper should be combined into a single paper which overall might tell a more comprehensive and compelling story.

(3) The paper does not generate testable predictions or practical insights that could be used by used by practitioners. The only takeaway point for me was that some neurons / units show correlated representations, which is arguably already known (e.g. Csordas et al 2020). How to exploit this modularity to develop human-interpretable explanations of network function remains unclear to me.
review_title: Worthwhile ideas and concept, but feels incomplete

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper explores the application of spectral clustering methods to assess modular organization in the emergent architecture of deep networks. In particular, sub-modules identified by spectral clustering are evaluated in terms of "importance" and "coherence", two metrics defined by the authors with the goal of capturing how crucial the neurons in the sub-module are to the classification accuracy, and how consistent their activation is across input and output patterns.
Overall, the paper addresses important questions related to the way structural properties in a deep network might support the emergence of functional properties, which is a key issue given the relatively poor theoretical understanding we have about these self-organizing systems. The paper is comprehensible, though the general structure and the writing could be improved to improve readability. For example, in Section 3 it is not immediately clear how the importance and coherence metrics relate to the specific technique adopted for feature visualization, or to the lesioning method applied. The “Related Work” section should be moved at the beginning of the paper, and the contribution should be better framed in the context of other existing approaches.
Although I appreciate the wide range of networks tested by the authors, I think that the way results are presented does not easily allow to establish whether we can identify a series of robust findings that are valid across all architectures / tasks, or whether specific cases entail peculiar findings. It would also be valuable to better assess the relationship between modularity and regularization techniques, such as dropout, L2 and pruning: I think that this is a very important point that should deserve further investigation, since it could give important insights about the role of regularizers in shaping the final network architecture.
Finally, it would strengthen the paper if the proposed method is put in relation to other approaches that have proven effective to address this kind of question. This would greatly improve the robustness of the results, since the current baseline is basically constituted by a comparison with random sub-modules. For example, it would be nice to see if clustering coefficient and average path length (as defined in [1]) can provide useful information also for the analyses proposed by the authors. Note that in [1] the authors investigated deep networks with similar architectures (MLPs, CNNs, ResNets) trained on similar tasks (CIFAR-10, ImageNet).

Other comments:
- Pg. 3: the technique used to visualize a sub-cluster by creating an aggregate measure of the learned features can be discussed in relation to the method based on Earth-mover distance proposed by [2], where the authors also discuss other graph-based metrics that might be useful in the present setting.
- Could the “intersection information” approach presented in [3] can be exploited also in the analyses of the sub-modules detected by the spectral clustering? Note that in [3] the authors also investigate “lesion tests” by means of interventional techniques, which would make that approach very interesting as a further benchmark.
- What is the rationale for setting the number of clusters to 12? If this value is not theoretically motivated, further analyses should show that the results are robust to variations in this value.
- When comparing “true sub-clusters” with “random sub-clusters”, a useful control analysis would be to create random sub-clusters by matching some connectivity property (e.g., same average strength, clustering coefficient and/or average path length).
- Regarding the gradient-based method discussed in Olah et al. (2017), it would be useful to have some dispersion measure over the final optimization score, in order to better assess whether all neurons in the sub-cluster where in fact similarly activated by the optimized image.
- It would be interesting to include as baseline some analysis on randomly connected networks, since it has been shown that subgraphs in large random networks can in fact support accurate task performance even without ever training the weight values [4].
- Where the p-values corrected for multiple comparisons?
- The authors consider “modularity  as an organizing principle to achieve mechanistic transparency”. Though I sympathize with this statement, I guess there are several cases where modular systems (or in general systems with localized representations) can develop complex emergent dynamics that still prevent interpretability.

References
[1]	J. You, J. Leskovec, K. He, and S. Xie, “Graph Structure of Neural Networks,” in International Conference on Machine Learning, 2020.
[2]	A. Testolin, M. Piccolini, and S. Suweis, “Deep learning systems as complex networks,” J. Complex Networks, vol. 0000, no. 1, pp. 1–21, Jun. 2019.
[3]	S. Panzeri, C. D. Harvey, E. Piasini, P. E. Latham, and T. Fellin, “Cracking the Neural Code for Sensory Perception by Combining Statistics, Intervention, and Behavior,” Neuron, vol. 93, no. 3, pp. 491–507, 2017.
[4]	V. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari, “What’s Hidden in a Randomly Weighted Neural Network?,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11893-11902. 2020.
review_title: Interesting proposal to inspect the emergent structure in deep networks, which should be refined and compared with alternative methods based on graph-theory

final_score: rating: 5: Marginally below acceptance threshold
review_content: The manuscript introduces an approach, based on importance and coherence, for evaluation whether a partitioning of a network exhibits modular characteristics. 
Importance refers to how crucial is a neuron , or set of neurons, to the performance of a network on a given task, e.g. classification.
Coherence refers to how consistently the neuron(s) in question are related to specific features.
Experiments are conducted by considering sets of neurons identified via a spectral clustering algorithm.

The manuscript proposes a method to verify to what extent a partitioning of a network follows modular characteristics. To a good extent the proposed method is grounded on proper theoretical foundations which is highly desirable. The only part where this cannot be fully verified is its dependence on the method from [Anonymous, 2021] which cannot be verified.

My main concerns with the manuscripts are the following:

- When conducting spectral clustering, the number of clusters is set to 12. Is there a procedure to set this value in a principled manner? is there an indication on the effect of this parameter? the manuscript would benefit from analyzing the effect of this parameter in the observation made on the reported experiments?

- Visualizations discussed in Sec. 3.1.1 (Fig.1) are quite subjective. While in some cases some patterns are indeed visible, in other cases it is hard to make sense of what is being presented. Is it possible to evaluate the produced visualizations in a more objective manner?
In recent years, several methods ( Bau et al, 2016, Oramas et al. ICLR'19, Yang and Kim, arXiv:1907.09701 ) for quantitative evaluation of methods for visual interpretation and explanation have been processed. Perhaps one of these could be adopted in the manuscript with the goal of objectively evaluating the visualizations/explanations presented in Fig. 1.

- In some cases design decision are made that seem to favor observations expected in some experiments. For instance, in order to favor clusterability small MLPs are pruned, to improve visualization MLPs are trained with dropout; and other factors relevant to the proposed method. Therefore my question by ensuring that some of these properties, e.g. cleaner cluster, clearer visualizations, don't you favor the measurement capabilities of the proposed importance/coherence metrics?

- When analyzing the "importance" metric on the lesion tests (Sec. 3.2) there are new conditions that are applied to the clusters being considered in the analysis, e.g. the size of the cluster, minimum effect of the cluster on accuracy, etc. Keeping this present, my questions are: i) Were these conditions also applied when analyzing "coherence", and ii) why these type of condition were not applied in the experiments of Sec. 3.1? Ideally, some level of consistency is expected among the experiments. Otherwise it is hard to assess properly the origin of observations made on the results of the experiments.

- At the end of Sec. 4, it is stated that the conducted experiments, combining spectral clustering with feature visualization,
highlight the usefulness of combining multiple interpretability methods in order to build an improved set of tools for rigorously understanding systems. However, from the observations made on the experiments I do not see the added value that the proposed method could bring to interpretability/explainability of the analyzed models (networks).

- Very related, one paragraph later, it is stated that having modular networks is useful both for interpretability and for building better models. However, from the content of the manuscript it is not clear how having a modular network/representation does contribute with the two listed aspects.

- Significant parts of the manuscript are delegated to the supplementary material. In addition, the third part of the proposed method, i.e. intrinsic partition evaluation, is part of another manuscript [Anonymous,2021] that does not seem to be published. For these reasons, to a good extent, the manuscript is not self-contained. 
review_title: Interesting work,  some aspects need to be polished

Meta-review: The paper present an approach for defending for, and search for, 'modularity' in neural networks, as a step to better interpretations of their functional structure. This is an interesting, and highly original approach, as recognised by the reviewers. However,  there was also some discussion about what exactly can be learned from the derived clusters/modules, and if and how they will lead to a better understanding of neural networks, or provide concrete ways of improving them.  While the authors addressed some issues during the review process, and provided additional results, the consensus (of all three reviewers) was finally that the paper did not reach the quality standards required by ICLR. I share this view-- the paper provides a refreshing perspective, but I still am not convinced that I see a clear, compelling 'use case' for their approach.  
Decision: Reject