Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes to prune CNN networks for each class in order to study interpretability of the networks. However, there are several significant drawbacks:

1) The pruning approach is too simplistic. Network pruning has been a field of very active research as the authors have acknowledged, however, the approach used in the paper is a very simplistic remove the ones with lowest response one, for which, there is no experiment to justify its validity w.r.t. state-of-the-art pruning approaches. In fact, from Fig. 2 and Fig. 3 one can almost conclude that this naive pruning method is significantly inferior to state-of-the-art.
2) Lack of novel insights. For an explanation paper, we would expect to obtain some new insights about the classification. The results that this paper get to, such as few filters were pruned in the lower layers, similar classes share similar filters, are not necessarily new knowledge to the community. In terms of the result analysis, mostly simple correlation analysis was used which presented no novelty nor insights.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
Summary
---

(motivation)
This paper proposes a new approach to pruning activations from neural networks,
but uses it to understand neural nets better rather than trying to make them
more efficient. It prunes filters per-class 1) to measure how sensitive image
classes are to pruning and 2) to measure how similar classes are by comparing
the filters they prune.

(approach)
Using only images from class c, CNN pre-activations are aggregated across spatial
dimensions and examples. This gives an average feature vector for that class.
T percent of neurons are pruned. Features with lower pre-activations
(possibly negative with large magnitude) are pruned first with successively
higher activations pruned later.
No re-training is performed.

(experiments)
Experiments use AlexNet and 50 of the 1000 ImageNet challenge classes to show:
1. Pruning filters results in decreased accuracy with smaller decreases in accuracy as the first and last few filters are pruned.
2. More filters from middle layers are pruned than are those from early and late layers.
3. After some filters have been pruned, sometimes pruning can increase accuracy.
4. I could not understand section 3.2.


Strengths
---

The proposed pruning method is simple and efficient.

I like the broad goal of understanding CNNs. We could use more papers that just focus on analysis like this one.

I think the idea of class conditional pruning is novel.


Weaknesses
---

# Major Weaknesses

I don't see why these results are significant. I do not find these results very surprising (see next comment) and I do not see why the community will find them useful.

* In particular, consider the conclusion. Sentences 2, 3, 5, and 6 seem to all be observations about what happened in the experiments. The conclusion should re-iterate the results, but it should also say why they are important. How does the work relate to the goals of the community at large? Which goals? Will this enable important new capabilities? What general concepts did we learn from this that we didn't know before?

Some of the positions in the paper could be more carefully considered. There are alternate explanations for many of these phenomena.

* Pruning smallest activations doesn't make sense to me. Typically redundant or un"important" activations are pruned because doing so has negligible impact on accuracy. The smallest activations are pruned here. Should the smallest activations be redundant or unimportant in some way? Does this rely on ReLU activations following the pre-activations? I think these values are not necessarily redundant and could be quite important (e.g., as measured by some saliency explanation like Integrated Gradients), but I could be wrong. It would be useful to provide a baseline which removes highest instead of lowest activations.

* Much of the surprise about figure 5 seems to be because it is not monotonic but it was expected to be monotonic. I agree that these curves should generally go down as theta increases, but I don't see why that relationship should be strict. I would be surprised if activations were not in some way dependent on one another. Furthermore, why does that dependence have to be interference? Couldn't it also be that some activations are complementary (and thus ineffective when only one is present)?

* Does Network Wise Pruning (NWP) favor more layers than others? It may be that some filters have higher Accumulated Responses per filter simply because there are more feature maps in the previous layer (thus more things summed up) and not because of what information they capture or their relationships with other filters. Does this happen? This could be an alternative to the following conclusion: "This means that the encoding provided by the first and last layer seems to be the most crucial and the densest."


# Other General Weaknesses:

* AlexNet is a rather old architecture to use for this analysis, so I can't be confident these methods or behaviors will generalize to other architectures. Does it hold for more modern architectures?


# Missing details / Points of confusion:

* The paper says the correlation from Figure 6 should be expected to decrease as theta increases. Why should this be expected?

* What exactly does Figure 6 measure? I think it's the ratio of the size of the intersection rho_sigma to the size of the union of the same two sets. However, the text calls the metric "correlation." This should be made clearer.


# Minor presentation weaknesses

Some parts of the notation/explanation don't make sense to me:
* "Let Lambda=... be the number of object classes in the dataset." But Lambda is defined as a set, not a numeral.
* The number p needs more context/subscripts as it depends on class c and filter i.
* "θ = [0, . . . , 1]" This defines theta as a finite set, but I think it's meant to be a number in the interval range (0, 1).
* Why is lambda_c needed instead of just using the index c to specify a class? The variable lambda doesn't seem to be any different than a class index.
* "let Fσ and Fσ ̄ be the set of unpruned and pruned filters for a given γ..." I don't see how this works. According to eq. 3 gamma depends on a particular filter i and class c, so the only filter F could contain is the ith one. (Later it became clear that F was only mean to class conditional, not filter conditional.)

Figure 5: These plots shoul all have the same y axis range. This would make them comparable and allow readers to much more easily compare trends across classes. Similar steps should be taken so the same range is used any time theta is plotted on the x axis.

Figure 3: I find it hard to get an overall ordering of the approaches in this figure because there is so much variance from class to class. It effectively conveys the variance, but I'd also like to know what the means across classes are for each method so I can compare the proposed approaches more effectively.

"The results indicate that classification classes are asymmetrically represented by filters resulting in the fact that some object classes have their classification accuracy increased when pruned for."
* I'm not sure what it means to be asymmetrically represented by filters.


Suggestions
---

This analysis would have been more interesting with an existing pruning approach because we would already know that such an approach is good a removing unnecessary filters.

Final Evaluation
---

Quality: Experiments could have been cleaner, but they basically demonstrate the patterns the paper intended to show.
Clarity: I could understand most experiments at a high level, but I found it hard to understand the motivation and the rest of the experiments.
Significance: As explained above, I do not see why the paper is significant.
Originality: The experiments and the proposed class conditional pruning approach are somewhat novel.

The paper is somewhat novel, but do not find it very clear and I do not see why it is important, so I cannot reccomend it for acceptance.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work proposes to prune filters in CNN model to interpret the correlation among different filters or classes. Though interpreting CNN filter is a well-studied topic, learning the interpretability through pruning is new and interesting. The proposed method is simple, by just using the averaging the value of the output of each filter as the indicator. The author claims that object classes represented in high feature density area usually share similar filters, which is in accordance with the common sense. Also, filters at lower layer are usually important. 

Some questions: 
1.	Since each filter is still like a black box, is it possible to visualize some result of the discovered interpretability?
2.	I’m confused with the implementation of pruning. If the filter at layer j-1 is pruned, then the dimensions of the filters at layer j should also change. How this issue is dealt with? Further, will this dimension reduction, instead of the pruned filter itself, influence the model performance?
3.	Why not use the absolute value of r_i? Any justification for this?
4.	The author mentioned that no normalization across categories are applied. However, are r_i from different layers comparable under NWP? Also, How can you guarantee that the Eq.(2) is comparable for different filters? More discussion on the normalization is desired.

review_title: Official Blind Review #2

Meta-review: The authors propose to use pruning to study/interpret learned CNNs. The reviewers believed the results were not surprising and/or had no practical relevance. Unlike in many cases, two of the reviewers acknowledged reading the rebuttals, but were unswayed.
Decision: Reject