Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel method to tell whether a unit (feature map) is effective in CNNs quantitatively, via "feature entropy". The proposed method builds a weighted graph based on a feature map and calculates the Betti curve. Then it defines the "birth point" that implies the appearance of regularized spatial pattern of notable components in the feature map. By computing the "birth point" based on a sample of images from a specified class, it can get a birth distribution, upon which the "feature entropy" is computed.
main_review: Strengths:
1. I enjoyed reading this paper. The proposed method is introduced clearly step-by-step with figures.
2. Main claims are well-supported. The motivation is explained in Section 1 by pointing out two flaws of previous indicators. And Section 4.3 shows how the proposed method address the flaws point-to-point through experiments.
3. The finding from Section 4.4 is interesting. The proposed indicator seems to be a good replacement for extra testing data. This would be useful in some real cases where labels are expensive and we want to use all the labeled data to train a model for the potentially best performance.

Weaknesses:
1. It would be good to repeat the experiments on ResNet. I'm curious about how skip connections affect "feature entropy".
2. Sampling 100 images seems to be too small. Any reason on the sampling size? Why not computing on all images?
3. Network pruning is mentioned in the related work. It would be interesting to see how the proposed method guide pruning.
4. Minor: It is not clear to me why the adjacency matrix needs to be symmetric. Thinking backwards, does this suggest symmetry in the feature map is important or not?
5. Minor: There is no $w_v$ in equation (3).
summary_of_the_review: Overall, it is a solid paper with clear introductions on the proposed method and good experiment design supporting the main claims. However, the fact that experiments are all based on one kind of CNNs is concerning. I would consider it as a board-line paper with the current version.

-------------
The author feedbacks are satisfying. The improvements on the paper with answers to Q1&3 are substantial. I've raised my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper applies well known methods in topological data analysis for quantitative assessment of neural net unit performance. Specifically, a scheme is proposed wherein spatial patterns of activation in the network units are determined as function of a sequence of activation thresholds, patterns transformed to a graph representations (i.e. successive filtrations) and their corresponding kth betti number, (i.e. number of kth dimensional holes), evolution calculated. In this paper k is set to 1.  For units that the authors term, effective units for a given class and a given input image class population, the analysis is expected to yield to similar birth time of the holes. Thus, a feature entropy is defined on the first birth time in the betti curves computed on a given instance and aggregating the estimated birth times over a population of instances in a given class.  The main results include the illustration of invariance of the calculated feature entropy when one does reweighting / rescaling transformations in the inputs and the illustration of correlation between the choice of hyperparameters, perturbations in data sets, and the feature entropy measure. 


main_review: The main essence of the paper is to translate the activations of the neural network for a given pattern class into a graph representation whose topology evolution is analyzed (using TDA) as a function of activation threshold.  The main novelty is the combination of well known methods  from topological data analysis and apply it to neural unit characterization.   The feature entropy measure defined seems to be more stable across variations do the rescaling and perturbations and the measure seems to correlate with network hyperparameter changes and perturbations. Thus the measure may be suitable for characterization of relationships between input/output in a systematic way.  

However, The main weakness of the paper is that the results in the paper are not conclusive on how this work will impact neural net design choices and optimization.  If this paper can be extended to address this, I believe the work will have impact. 

It is not clear to me why the authors take the activations and perform a symmetry transformation (A^v = max (A_v, (A_v)^t) ) after equation 3.   It is not clear to my why you chose 0.1 for p in equation (12).  What is the rationale here?  

The experimental sections can be improved in clarity. While I follow the general results, I cannot completely comprehend the fact that the feature entropy drops rapidly and the value in the last layer is large.  Can you elaborate about this and the above points in the rebuttal?

While the ideas in the paper are clear, the writing needs to be improved.  There are several places where the sentence structure is awkward and it is hard to comprehend what the authors state. 
e.g. 1)  In contrast, for ineffective units, since being incapability of effectively  ---> change to:   being incapable of effectively
2) In Hofer et al. (2017), topological signatures of data are evaluated and used to improved
the classification of shapes. --> change to: used to improve
3) rewrite line in section 3.2 -- "For a specific target class C, consider image I(i) ...  By perceived
with an ideal unit U, it should present similar pattern with other image samples."
4) section 4.2 -- when referring to figures and results the sentences are often awkward to the reader..
5) section 4.3 -- "Since random units are clearly incapable to well perceive features
like those trained units,"   should be changed to 'perceive features well'
6) section 4.3 -- reference to ApoZ seems to be wrong (As far as I understand ApoZ was proposed by Hu et al, not He et al).



summary_of_the_review: Overall, I think the paper is interesting and the ideas are worth pursuing. The writing needs significant improvement, the experiments are still limited.  I am not sure also not sure about the impact of the paper, in its current form, and its utility for enabling improvements in design of neural net architectures.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a topology-based approach to measure the effectiveness of a unit in a neural network for a set of inputs (i.e. images from a specific class). A graph is constructed based on a given image and the units representation and the birth point is computed. The importance of a unit is then measured as the entropy over the birth point distribution, which is estimated by sampling a set of images. Experimental results highlight the methods advantage over three baseline approaches and illustrate its ability to discriminate models with different generalisation capability.
main_review: Strengths: 
1. The paper is easy to follow and the authors clearly highlight the problems with prior approaches when measuring the importance of a unit.
2. The experimental analysis corroborates these problems nicely and shows that they can be solved by the proposed feature entropy approach.

Concerns:
1. The advantage of feature entropy over the three baseline approaches is illustrated by conducting two tests (rescaling and randomness) and observing the behaviours. However, the comparisons mostly represent a sanity-check and do not illustrate the importance and practical benefit of the proposed approach. 
2. Previous approaches that measure the importance of a given node have often been compared in pruning scenarios. Is there a reason, why feature entropy can not be used in this setting?
3. The method is compared to relatively old baselines. For instance, [1] and [2]  address a similar task of measuring neutron importance (in a pruning setting).

Minor:
- It is not quite clear from the start, what is meant my “status” of a unit. Maybe “importance” of a unit will be clearer.
- Consider adding Model BA in Table 4 to improve clarity.

[1] R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis. NISP: Pruning networks using neuron importance score propagation. In CVPR, 2018.
[2] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In CVPR, 2019.
summary_of_the_review: Overall, the paper is interesting and the proposed approach of measuring unit importance is well motivated. However, as mentioned above, I do have some concerns regarding the empirical evaluation of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a technique to measure the state of each neural network unit. The features of the neural network units are given by a method that applies TDA to a graph representation of the activity degree of each unit. The authors also experimentally measure the degree of learning and show its effectiveness against other methods in terms of scaling and randomness.
main_review: Strength
- The establishment of a technique for estimating the state of a neural network with respect to training will be useful for various applications, and the creation of a technique that can check the progress of training and compare trained NNs is expected to contribute to the development of AI.
- Such measurement techniques are generally plagued by scaling and randomness problems, but the entropy-based approach is a significant contribution in that it solves these problems.
- Being able to analyze the learning progress at the structural level of the NN can also provide information for ways to improve performance.
- Although it is a natural idea to measure the distribution of the activity of each unit, it is an interesting idea to construct it on a TDA basis, because a simple measurement method has many problems.
- Although the proposed method is heuristic and still in the observation stage, it is worthy of further development.

Weakness
- The authors state that the size of the unit can be assumed without loss of generality, but that seems to be a rather strong assumption. Since the proposed method can be computed only for square matrices, it cannot be applied directly when the number of elements in the data or filter is not square.
- It is also questionable that the unit is two-dimensional. In general, the middle layer of a CNN is multiple images. One of the advantages of the proposed method is that it can be measured independently of the order of the elements (i.e., the numbering of i and j). This is because the order may change in each training trial. The single-sheet assumption may not be a problem if you compare each corresponding sheet, but the order of the sheets will be switched in each trial, making it less useful.
- In TDA, we generally refer to the time of occurrence and disappearance of the simplex in filtration as birth time and death time, respectively, so calling the first non-zero state of the Betti curve the birth point may cause confusion.
- The reasons for using birth points are explained in the appendix. However, if, for example, another 1-cycle is created immediately after the first 1-cycle is created, it means that significant activity has taken place in a different region, so ignoring it would be lacking critical information. The trade-off between ease of calculation and lack of information will need to be discussed.
- Each cell of a CNN unit is fundamentally independent and does not represent any relationship. It may retain the positional relationships of the input images, but it is still counter-intuitive to represent the activity of the units in a relational graph. Therefore, it is difficult to understand why this construction method extracts the features of the activity of the unit.
- This construction method may only extract information similar to the change in the number of active cells for a change in threshold. In this case, there is no effect of using TDA. These things need to be considered and verified in order to show that the proposed method makes sense.
summary_of_the_review: The effects suggested by the proposed method are very useful and can be considered a sufficient contribution. On the other hand, the proposed method is heuristic and limited, and there are some questions about the theory behind feature extraction, which require deeper investigation in the future.

[Additional comments after discussion]
Through discussion and revision, some of the unclear points were clarified and the paper became better. The proposed tool has the potential to be a very useful tool and its contribution is appreciated. On the other hand, this is something that needs to be developed in the future. These points were taken into account at the beginning, and although they are relatively positive, the score remains unchanged.
review_title: null

Meta-review: This paper proposes a new method for understanding the role and importance of individual units in convolutional neural networks. The reviewers were in agreement that the technique is novel and provides potentially valuable insights into neural network behavior. The reviewers were less certain about the utility or significance of this idea; however, the authors partially addressed this concern by adding studies of using this technique as a pruning heuristic, and future researchers will be the best judge of the paper's eventual significance. With that in mind, I recommend acceptance so that this intriguing idea can become part of the research literature and future researchers will have this opportunity.
Decision: Accept (Poster)