Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a structured pruning method which turns structure pruning into a graph pruning problem.  The authors represent each input, output pair as an operator node, with edges measured by operator norms between operators.  The authors then propose an iterative structured pruning algorithm which prunes layers based on the longest path in the graph.  They then evaluate their method on three image segmentation tasks.
main_review: The method is simple in its approach, which allows the method to be easy to understand.  Suggestions for improvement are minor: Comparisons to additional methods presented in the related work and corresponding benchmarks would strengthen the results (e.g. ResNet per Dong & Yang 2019). Benchmarks for the full network could help better contextualize pruned results. Replicates are presented in a way that is difficult to distinguish methods from one another.  For example it's difficult to distinguish light blue vs gray lines in Figure 3.  Additionally, variability of speedup between training runs is not presented in figure 4. Moreover, the relative number of remaining convolutions metric is presented a bit vaguely.  It's not clear why some panels have these metrics missing.  Additionally, it's not clear how one would calculate what accuracy is used to calculate "relative number of convolutions at equal accuracy" for each graph.
summary_of_the_review: Overall, the method is easy to understand and the paper is clearly presented.  Some organizational and presentation changes could help make the results easier to interpret.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes the LongEst-chAiN (LEAN) method to perform structured pruning of CNN networks. LEAN maps a CNN network to a pruning graph, where every channel of input/output is a node, and every operator is an edge connecting input and output nodes. It uses the operator norms as the weights of edges. Then it prunes the network by keeping the longest path in the graph iteratively until it reaches the target pruning ratio. This paper demonstrates the effectiveness of LEAN pruning by comparing it to two structured pruning methods (structured magnitude pruning, operator norm pruning) across three image segmentation datasets (Simulated Circle-Square dataset, CamVid, Real-world dynamic CT dataset) and three CNN architectures (MS-D, U-Net4, and ResNet50). Experiment results show that LEAN outperforms the other two structured pruning methods as it achieves similar model qualities with much smaller pruning ratios in most cases. Also, the paper shows that a MS-D model pruned with LEAN is 10.9X faster than the unpruned network in practice.
main_review: Strengths:
1. The idea of using a graph-based algorithm for structured pruning is novel and interesting. Since LEAN prunes the network by keeping the longest path in the graph, it can use the global pruning ratio without worrying about generating a disconnected network. Therefore, it may prune much more aggressively in some layers than the other two structured pruning methods.
2. Experiment results show great potentials for LEAN pruning.

Weaknesses:
1. A primary concern is on the evaluation. (a) The paper only compares LEAN with two naive structured pruning strategies, but not other more advanced methods. (b) The datasets used are too small (e.g., 1000 training examples in the Simulated CS dataset). It is not clear whether applying ResNet50 or U-Net4 is reasonable. Why not use those commonly used datasets (e.g., Cifar10, ImageNet) and networks for evaluation? (c) The speedup results are only shown for an MS-D network on the dynamic X-ray CT dataset. And the paper mentioned that it uses a custom MS-D model which loads only the unpruned filters. Then I wonder what the speedups of networks pruned with other structured pruning methods or unstructured pruning methods are with such customized implementations.
2. I would like to see some theoretical analysis on why LEAN outperforms the basic structured pruning methods such as structured magnitude pruning or operator norm pruning. It may help make the paper more rigorous.
3. The authors may want to improve the presentation to better describe the definition of preliminaries and how to construct the pruning graph. Figure 1 (B) is confusing. 


Minor comments:
1. The figures are not black/white printer friendly.
2. Most of the related works discussed in section 2 are those before 2020. I would like to see more discussions on the most recent related pruning works.
3. It might be better also to report the accuracy/F1-score/mIoU of the unpruned network in the paper.
4. It would be good to see the structure of the pruned networks with LEAN in either the main paper or the appendix.
5. In the last paragraph of section 6.1, the paper mentions structured magnitude pruning performs slightly better than LEAN for a few pruning ratios with U-Net4 and the CamVid dataset. But it doesn't give any explanation or discussion.
summary_of_the_review: Overall, the idea of the paper is novel and interesting, and the experiment results look promising. However, the paper fails to compare LEAN with the most advanced related works on structured pruning. The datasets used in the experiments are too small, and the experimented tasks are not commonly used in CNN pruning works. So it is hard to justify the advances of the proposed pruning method. Also, the presentation of the paper needs to be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper abstracts the AI model as DAG and proposes to use the longest chain path with accumulated operator norm as criteria to perform a greedy pruning method. 
main_review: Strengths:
Propose the new concept of the longest chain path over DAG, which is used in model pruning. 
 
Weakness:
1.  The algorithm is actually in a greedy manner to remove the path one by one. Actually, in previous existing work, there are some discussions about how to deal with the wrongly pruned channels at the early stages of pruning. Any discussion about this issue? 
2.  Any comparisons on classification benchmarks on imagenet? I think as a model pruning method, it is necessary to compare with other many SOTA pruning methods on classification benchmarks, to demonstrate its superiority. 
3. For image segmentation tasks mainly discussed in the paper, the comparing methods are the only magnitude and operator norm, which I think is also not enough. 
summary_of_the_review: Due to the aforementioned weakness, I temporarily think this paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a structured pruning approach by building a graph according to the structure and weights of the CNN to be pruned. The operators with the longest chain in the graph as preserved while others are pruned. The proposed approach is evaluated with several network structures (MS-D, U-Net, ResNet) and datasets.
main_review: Pros:
1. The proposed approach Use graph-based concepts for network pruning, which is a new approach that raises researchers' interest recently.
2. The proposed approach can prune the network with large pruning ratios.
3. The proposed approach works on segmentation tasks.

Cons:
1. The introduction and related work sections are not described well.

    The motivation of the proposed methods is not described well. Avoiding disconnectness seems to be a weak reason. The related work section lists a number of previous pruning works but it seems difficult to connect them with the proposed approach. On the other hand, related graph-based pruning approaches are missing. For example, [1] proposes to build a graph for each conv layer and prune filters based on the graph complexity. 

    [1] Convolutional neural network pruning with structural redundancy reduction. In CVPR 2021.


2. The proposed approach lacks theoretical justification. It is not clear why extracting the longest chain in the graph is the optimal choice for filter pruning.


3. The methodology is not presented well and is hard to follow. It is presented narratively, without formal definitions of the concepts used in the approach and equations, making it difficult to understand the implementation details. Specifically,

    -In Fig. 1(B), what do the blue line, (i) and (ii) mean?

    -It is hard to distinguish the concepts of operation and operator. According to Section 4.1, a node is a channel and an edge is an operator. However, in Fig. 1(B) it seems that a node represents the output of a filter rather than a channel?

    -I think the calculation of operator norms is very important and should be moved from the Appendix to the main text. (On the other hand, the description of datasets in Section 5 is trivial.)

    -It is unclear why $||AB||  \approx ||A||\cdot||B||$.

    -It is unclear how to search for the longest path in the graph.


4. The experiment settings and results are not satisfactory.

    The proposed approach is not evaluated with benchmark classification tasks such as CIFAR and ImageNet. Besides, only structured magnitude and operator norm pruning are used for performance comparison. Both approaches are proposed many years ago. State-of-the-art methods should be added for comparison.
summary_of_the_review: Overall, I think there are several nontrivial weaknesses in the manuscript as listed above. I currently give a negative score but I will consider changing it if the authors' feedback resolves the concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a pruning method for CNNs by using a graph-based algorithm. It was evaluated by using CS, CamVid, and dynamic CT dataset.
main_review: The idea is somewhat novel and sound. It is proved to be effective in the evaluation.
summary_of_the_review: The size of the databases used for the evaluation seems rather small. Hence, the improvement may be obtained from the adaptation of models to the target domain. It would be better to evaluate this method by using datasets with a larger size to clarify this point. Also the comparison with the existing pruning methods are not sufficient.
review_title: null

Meta-review: I do not recommend accepting this paper, although I make this decision with reservations. The review quality for this paper was not particularly strong, and I wish to emphasize to the authors that I read the paper myself in detail in the process of writing this metareview.

This paper proposes a new structured pruning technique called LEAN. It involves computing an operator norm of the convolutions in a convolutional neural network, multiplying these norms over paths through the network, and keeping the paths with the highest such values (and pruning everything else). This paper makes the argument that this metric is robust to scaling and prevents network discontinuities. (In this way, the technique is very reminiscent of SynFlow (Tanaka et al., Pruning Neural Networks without any data by iteratively conserving synaptic flow) in terms of motivation and resulting technique, although SynFlow is unstructured. I do not mean this as a criticism - just a suggestion for the authors of a connection they might be able to make in the future.)

One big concern I have about this paper based on the methodology alone is as follows: the paper states a number of hypotheses about why this is a sensible way to prune (e.g., in the beginning of Section 4). I see no reason why any of these hypotheses are wrong, but the paper never makes an effort to evaluate any of them. I don't mean a theoretical justification here - that's difficult and unlikely to yield useful information about what happens in practice. I mean experiments to ablate the salient properties of the heuristic mentioned in the paragraph at the beginning of Section 4 (Does scaling invariance actually matter in practice? Is network disconnectivity actually a risk in practice?).

My biggest concerns about the paper, however, are in the evaluation. I share two major reviewer concerns that were mentioned:
(1) The paper compares to a very limited set of baseline pruning methods, and relatively older ones at that (2019 is indeed old in the world of pruning).
(2) The paper does not look at standard, "large-scale" benchmarks for computer vision - namely, ResNet-50 on ImageNet.

Neither of these concerns is necessarily decisive in my view. For example, with respect to Concern 1, the reviewers unhelpfully do not suggest very many additional structured pruning benchmarks to consider, and I think the additional baselines added during the revision process have softened this concern. I would also recommend taking a look at "Growing Efficient Deep Networks by Structured Sparsification" (Yuan et al) for a useful method and a good set of baselines. There are an arbitrary number of baselines one could add and the structured pruning space is a confusing mess, but I think the claims in this paper merit more than are currently present.

With respect to Concern 2, I'm even more conflicted. On the one hand, I have rarely seen any pruning techniques proposed for or evaluated on vision tasks beyond image classification, despite the fact that - in the real world - segmentation is much more popular than it would seem by reading the ICLR proceedings. To that end, I applaud the authors for focusing on those settings and I see substantial value in a paper that does so. On the other hand, ResNet-50 on ImageNet (among other standard classification benchmarks) is the de facto measuring stick for evaluating pruning methods in computer vision, and the exclusive focus on segmentation here means it is very difficult to compare the proposed technique to other benchmarks. If the paper is to focus on segmentation alone, this places a higher burden on adding many additional comparisons to other methods (i.e., Concern 1). Finally, I don't see any reason why the paper *couldn't* also include ResNet-50 on ImageNet or the like in addition to segmentation; if it is a limitation on the compute available to the authors (something I empathize with), they did not say so in any of the author responses. Upon reading the author responses, I was left asking, "Why not both?"

For those reasons, I do not recommend accepting the paper, although I think there are some good reasons to value the paper's contributions. At the end of the day, there are some relatively simple things that could be changed to make the paper much easier to contextualize within the pruning literature. As of right now, it would be very difficult for me to say whether or in what contexts this method should actually be used in practice.

(P.S. I agree with the reviewers that Figure 3 is exceptionally hard to parse.)
Decision: Reject