Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Dynamic isometry is shown to be a useful property that enable effective gradient propagation through the forward/backward. However, pruning will largely damage such a structure. This paper studies how to maintain the “dynamic isometry” property during pruning. Specifically, after getting an initial assessment of filter importance, the algorithm will maintain the partially kernel orthogonality of the important filters. They also propose to regularize the BN parameters to future boost the performance.
main_review: Pro:
The authors propose a nice way to maintain dynamic isometry property during pruning: just doing a back-propagation considering the orthogonal constraint. The method is quite intuitive and simple, which I think it’s good.

Cons:
I think the imagenet performance is not quite better than previous methods. The improvement is in general 0.1 points except ResNet50 (3.06x) where the improvement is 0.5 (still not significant). If the authors want to claim their method has larger advantage in larger pruning ratio, there should be more experiments to support such a claim.

The authors argue to use “difference between performance under different learning rate” as a metric to measure whether dynamic isometry is preserved, which in my opinion is not a good measure. Can the author measure the jacobian of each layer?

Questions:

About the claim: “SGD training in finetuning can help recover it”, I’m confused what is the goal of recovery when looking at figure 2. The model seems not to converge to JSV 1, then what information this figure conveys?

What is the L1+KernOrth in Table 1 differs from the proposed method? Is the difference just whether regularize BN parameter? Moreover, there is no explanation of what is the difference between L1 + X v.s. X + L1, which is confusing.

Why this paper only use kernel orthogonality instead of convolutional orthogonality?

Comments:

I think the content would be much easier to read if you can create a 2D plot with the x-axis being the speedup and the y-axis being the accuracy. In its current form, all I can see is the proposed method does not have a clear advantage over the previous method.

Regarding maintaining approximately dynamic isometry, this paper is also related:
[1] deep isometric learning for visual recognition. ICML 2020.
summary_of_the_review: I like the idea of maintaining dynamic isometry during pruning. But there is still great room to improve in both the presentation and analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a dynamical isometry preserving approach for structured pruning by introducing an orthogonality preserving regularization for weight matrices in each layer. The experiments on cifar and imagenet datasets show improvements over comparable methods.
main_review: # Strengths

1. Orthogonality preserving structured pruning is interesting.
2. The results demonstrate the merits of the method.
3. Overall the paper is well written.

# Weaknesses

1. The role of BN regularization is not clear to me. It is not clear how the BN parameters of pruned filters would make a difference in dynamical isometry (DI) and how the proposed regularization alleviates this issue? More clarity is required.

2. It is not clear that the method indeed improves DI. Considering Fig.2 c, mean JSV increases and is higher than 1. So not clear the method indeed improves DI. 

3. The notation inside the Frobenius norm is not clear in Eq.3. Probably a type, please comment.
summary_of_the_review: Overall the paper is interesting. The connection to BN regularization and DI could be explained better to improve clarity.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a structured pruning method based on kernel orthogonality analysis. The method is straightforward and well-investigated in the network pruning field.
main_review: Pros:
1. Clear motivation and writing.
2. Well-established pipeline.

Cons:
1. Comparisons with the state-of-the-art methods are insufficient, such as [1].
2. Experiments on ImageNet are insufficient and results are not comparable.
3. Moderate novelty. This kernel orthogonality idea has been employed in deep networks in previous methods.


[1] ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting, Ding et al., ICCV 2021
summary_of_the_review: This paper has moderate novelty with the well-studied kernel orthogonality method. Besides, the experiment design is not sufficient. The performance gain in ImageNet is incremental and not sufficient compared to the state-of-the-art methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes regularizing learnable parameters of NNs to maintain the dynamical isometry during pruning and improve their accuracy. In the experimental analyses, the proposed OPP outperforms baseline methods on benchmark datasets.

The paper is well written in general, and the proposed methods are intuitive. In the experimental analyses, the proposed methods outperform baseline on various benchmark datasets.

main_review: The paper is well written in general, and the proposed methods are intuitive. In the experimental analyses, the proposed methods outperform baseline on various benchmark datasets.

However, there are few major issues with the paper:

- There are three main claims in the paper; achieving structured pruning, orthogonality and dynamic isometry. However, these claims are not explored in detail;

-- Structured pruning is only mentioned as a synonym for filter pruning or channel pruning in the review of the related work. This is achieved in the proposed method by user defined parameters that identify the indices of parameters to be pruned. In the analyses, selection of this set of indices affects the accuracy as well. That is, there is not a major contribution for structured pruning.

-- Orthogonality is aimed to be achieved by regularization on gram matrices of kernels. However, how well/much orthogonalization is obtained is not analyzed. In addition, regularization of batch normalization parameters has much more effect on the accuracy compared to orthogonalization. 

-- The paper claims to achieve dynamic isometry by orht. However, it is also stated “We also propose to regularize batch-normalization parameters for better preserving dynamical isometry for the whole network.” That is, either orth. does not enable dynamic isometry by itself, or dynamic isometry does not affect the accuracy remarkably. This claim should be examined theoretically or experimentally in detail.

To summarize, the relationship between these three proposed major claims should be analyzed in detail. Otherwise, the scope and claims of the paper should be updated accordingly.

Additional questions:

-	How did you calculate speedup? Could you please provide an example of actual running times for training and test for an experiments, such for the VGG19 + Cifar100 case, in comparison with the baseline?

-	Have you employed regularization of parameters of BN of baseline methods to check how this method also affects accuracy of baseline?

summary_of_the_review: The paper is well written in general, and the proposed methods are intuitive. However, the proposed claims are not explored well. More precisely, the relationship between the proposed major claims should be analyzed in detail. Otherwise, the scope and claims of the paper should be updated accordingly. To this end, the paper proposes regularizing learnable parameters of NNs using different heuristics. Therefore, the novelty and scope can be redefined according to provided analyses and results.
review_title: null

Meta-review: The paper proposes a pruning approach that regularizes the gram matrix of convolutional kernels to encourage kernel orthogonality among the important filters meanwhile driving the unimportant weights towards zero. While the reviewers found the proposed method well-motivated and intuitive, they believe that the proposed claims are of limited novelty and are not supported well by the experiments. Analyzing and explaining the effect of different parts of the proposed method, i.e., orthogonalization and regularization of batch normalization parameters, on the accuracy of the pruned models would significantly improve the manuscript.
Decision: Reject