Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors studied and formalized the effect of initialization to connection-sensitivity-based pruning. The authors first pointed out that a previously studied pruning criterion -- connection sensitivity (CS) -- is a normalized magnitude of gradients. Based on signal propagation theory, to achieve a 'faithful' (with minimal amplification) CS, the gradients must be also faithful. Then by using relation of Jacobians and gradient, the authors proved that orthogonally initial weights guarantees faithful on linear networks and certain distribution property on nonlinear network can achieve layerwise dynamic isometry, which is to ensure faithful signal propagation. Based on these findings, the authors proposed an initialization setup for improving pruning performance, with the goal to ensure dynamic isometry by orthogonal initialization and approximation.

In summary, this paper used a typical pruning criterion (CS) and used signal propagation theory to studied how initialization effect CS. Although this linking is quite limited, this paper has marched a step in quantitative and theoretical study of initialization in pruning: how pruning affects network performance and why it is. 

Here is a question:
In Fig.2, nonlinear activation is only shown with tanh. How is training performance when relu is used since relu is more commonly used in modern deep architectures.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper analyzes how signals propagate through randomly initialized neural networks that have undergone a kind of pruning/sparsification. The pruning method utilizes a metric called 'connection sensitivity', which has been used in prior work and which measures the infinitessimal impact of turning off specific parameters. The distribution of singular values in the layer-to-layer Jacobian matrices for pruned networks becomes increasingly pathological as the depth increases. This observation motivates the concept of 'layerwise dynamical isometry' (LDI), a slight generalization of the concept of 'dynamical isometry' that has been studied in prior work. Several methods for approximately obtaining differing amounts of LDI are investigated in a series of in-depth experiments that show a strong correlation between increased signal propagation and improved trainability of sparse networks.

Although there have been numerous works studying dynamical isometry as a principle for initializing very deep networks, and many other works studying pruning methods after (and recently before) training, as far as I'm aware there has been no prior work that examines the intersection of these two directions. As such, I found the contributions of this paper to be novel and believe the results will be of interest to practitioners and theorists alike.

An important contribution of this paper is in identifying that a main difficulty in pruning networks at initialization comes from degradation of signal propagation, leading to poor or impossible training. The numerous well-thought-out experiments provide compelling evidence that the trainability of pruned networks is highly correlated with spectral measures of the networks' Jacobians. 

The authors take this observation a step further by introducing a method for correcting the poor conditioning that can result from pruning. They show that enforcing Approximate Isometry on weights of the pruned connectivity pattern enables the pruned models to train much faster and often achieve better performance.

Finally, the authors look at two natural extensions of their analysis  to designing new high-performing architectures to situations where labels are not present. Overall, I found this paper to have a detailed and thorough experimental analysis and to present nice new perspectives on pruning and signal propagation.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper introduces a signal propagation perspective for single-shot network pruning. Particularly, by achieving layerwise dynamical isometry with proper initialization scheme, the paper ensure the reliability of connection sensitivity pruning criteria, thereby improving the pruning results.

I’m inclined to reject this paper because (1) methods of dynamical isometry typically care about trainability of very deep network (e.g., 10,000 layers) and it has little to do with generalization performance which is what we really care about in network pruning; (2) experiments on signal propagation were done in small datasets and fully-connected layers without batch norm, which is not the standard setting we conduct network pruning. It’s unclear whether the analysis would carry over to CNNs with batch norm.

Main argument:
This paper attacks single-shot pruning problem from the perspective of signal propagation. By using special initialization scheme, the authors attempt to calibrate the connection sensitivity measurements across layers. The main argument in the paper of dynamical isometry doesn’t has a direct connection with performance of a pruned network since it basically concerns the trainability of very deep neural networks.

Moreover, adopting orthogonal initialization satisfying layerwise dynamical isometry only leads to marginal improvements over standard variance scaling initialization, as shown in Table 2 and 3.

In terms of potential improvements, I would suggest the authors to first carefully study the relationship between trainability and generalization and also analyze signal propagation of networks with batch norm. Finally, for such an empirical paper, I expect more large-scale experiments conducted.

review_title: Official Blind Review #2

Meta-review: This is a strong submission, and I recommend acceptance. The idea is an elegant one: sparsify a network at initialization using a distribution that achieves approximate orthogonality of the Jacobian for each layer. This is well motivated by dynamical isometry theory, and should imply good performance of the pruned network to the extent that the training dynamics are explainable in terms of a linearization around the initial weights. The paper is very well written, and all design decisions are clearly motivated. The experiments are careful, and cleanly demonstrate the effectiveness of the technique. The one shortcoming is that the experiments don't use state-of-the-art modern architectures, even thought that ought to have been easy to try. The architectures differ in ways that could impact the results, so it's not clear to what extent the same principles describe SOTA neural nets. Still, this is overall a very strong submission, and will be of interest to a lot of researchers at the conference.


Decision: Accept (Spotlight)