Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors make use of a theoretical framework for viewing neural networks as representation of quivers to introduce a reparametrization strategy for neural networks with radial activation functions. This reparametrization is based on a QR decomposition, and leads to a lossless compression of the number of parameters by factoring redundant symmetries of the model. A corresponding gradient descent algorithm is derived on the compressed parameter space corresponding to gradient descent in the original space.
main_review: Overall, I found the paper well-written, and presenting an seemingly interesting observation on neural networks whose activation function commutes with change of basis (orthogonal linear maps). However, I have serious concerns regarding the “radial” assumption on the non-linearity, and I was not convinced by the utility of the introduction of an elaborate mathematical framework to demonstrate a phenomenon which can simply (and plainly) be explained through some standard linear algebra. I believe that the authors could greatly improve the paper by either 1) demonstrating some further insight from the view of the network as a representation of a quiver, or 2) writing the paper in straightforward fashion through standard linear algebra arguments. Additionally, I feel that the overall results in this paper are not surprising (given the assumption on the activation functions), and I fail to see how such a phenomenon may generalize to typical activations encountered in practice. Detailed comments can be found below.

**On the “radial activation” assumption and generalizing to further activation functions.**
The authors introduce an assumption on the activation function that it is “radial”. At first glance, this seems to me to render the network essentially linear up to a scaling factor. Indeed, the radial assumption means that $\sigma(x) = ax$ for some scalar $a$ (which may depend on $x$), and thus for a two layer with weights $W_1, W_2$ we have $W_2 \sigma(W_1 x) = a W_2 W_1 x$, and the network is thus a deep linear networks with some (scalar) scaling factor. Given this fact, it is not surprising that a reduced representation is possible, but this identity also significantly jeopardizes the expressive power of the network. The authors indicate that they expect their work to extend “non-radial activation functions”. I believe that such an extension is essential given the strong limits of the radial assumption. However, it also appears to me that the assumption that the activation commutes with orthogonal linear transforms is central to the proposed procedure, which places strong constraints on the expressivity of the network and does not seem generalizable to networks used in practice.

**On the mathematical formalism and arguments.**
The authors make use of a fairly new formalism of viewing neural networks as representation of quivers. However, it is not clear to me that this view provides any specific insight into the claims of the paper, and it seems that it would only hinder accessibility of the results to a wider audience. Indeed, I believe that the claimed result can be obtained (in a very similar fashion) through a proof by induction (on the number of layers) and standard linear algebra arguments. Given this, I believe that the authors should either leverage the proposed formalism to obtain insights specific to that formalism, or abandon the formalism altogether.

**On the gradient algorithm.**
In addition to the reduced representation, the authors show that projected gradient descent as applied to the reduced representation is equivalent to gradient descent in the original space, due to the fact that the gradient commutes with orthogonal transforms. However, this is not the case for many methods used in practice (e.g. Adam, RMSprop etc.), which explicitly depend on the choice of coordinates as they compute axis-aligned normalization quantities. It would be helpful for the authors to address these practicalities if possible.

Typos: p. 3: Clebsh-Gordan
summary_of_the_review: The paper presents some potentially interesting ideas. However, I believe that the unnecessary usage of an esoteric formalism and the constraining assumptions do not warrant publication at this time.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides a framework to study compression of NN models by analyses of symmetry properties of their parameter spaces. For lossless model compression, an algorithm employing QR decomposition for parameter compression is proposed. In the theoretical analyses, the decomposition of the proposed neural quiver is studied. The proposed methods were experimentally analyses in synthetic datasets.
main_review: The proposed neural quiver is a nice application of matrix representations in NNs, which have been studied in the literature as reviewed in the paper. The proposed paper employs them to study compression of models. However, the proposed theoretical and experimental results should be improved to justify the proposed main claim, which is proposing a framework for efficient model compression. 

To sum up, there are two major issues with the paper:

First, the proposed main theoretical results have been studied in various previous works for training NNs on Stiefel or low-rank matrices. In addition, additional works employed more efficient retractions (e.g. Cholesky decomp.) than QR decomposition for optimization (e.g. projected or Riemannian GD) on low-rank matrices or orthonormal matrices. Therefore, the theoretical novelty of the proposed method is limited.

Second, the main claim (also the title) of the paper is about model compression. Therefore, the proposed methods should be compared with the other model compression methods (at least, with those which employ PGD on low-rank parameters as proposed in this paper) experimentally on benchmark datasets in detail. 

Some minor comments:

In Fig. 1, is the bias shared among layers? If it is, then does the proposed neural quiver represent a particular class of NNs?

Please define R in Theorem 1. How do you compress W to obtain R?


summary_of_the_review: The paper proposes a theoretical framework to study compression of NNs employing the representation theory of quivers. However, the proposed theory and its experimental analyses are incomplete.

To improve theoretical results, I recommend first explicating the theoretical results in comparison with the related work. Second, experimental analyses should be extended by applying the theoretical framework for the target model compression tasks in comparison with the related state-of-the-art.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a dimensionality reduction technique that can preserve the model of a neural network by leveraging radial symmetry. Consequently, when the required conditions apply, they also show that training the compressed model is equivalent to training the original model on a particular projection of the parameter space.

The first two sections of the paper are very clear and well written, although some definitions would have helped (more on that later), but the sudden change of pace starting at section 3 makes it difficult to follow if you do not have the same background. I believe that I have captured the gist of their results, which seem to be intuitively correct, but I cannot comment on how they were derived. Hence, what follows is a description in my own words and perspective of what they did.

The authors present a method that would reduce the width of each layer of the neural network to the smallest width of any preceding layer, including the input layer. That is possible in radial neural networks because (1) the activation function involves the entire layer as opposed to each neuron individually and (2) the activation function is symmetric along any direction considered. 
main_review: # Comments on novelty and related work

What the authors regard as reduced widths, which represent the dimensions of a radial neural network after compression, has an evident relationship to dimensionality loss of the affine transformations along the neural network. I believe that the authors should point out at related work on neural networks regarding the bottleneck effect on the outputs caused by the width and number of activated ReLUs [1].

The paper somewhat implies this work as the first lossless compression approach of neural networks: "leading to a lossless model compression algorithm [...] Whereas previous approaches to model compression are based on weight pruning, quantization, matrix factorization, or knowledge distillation." However, there were three papers published on the topic in the last couple of years [2-4].

The comment regarding the classic LTH paper is inaccurate: "Recent work has shown pruning can be done before training by identifying certain key network structures (Frankle & Carbin, 2018)." In fact, that first LTH paper does pruning while training by (1) doing a few steps of training, (2) identifying lesser relevant weights, and (3) rewinding to the original parameters while excluding those parameters, which is not the same thing as pruning before training. However, the authors may (and perhaps should) cite subsequent work that does prune before training [5-8].

As a survey, Blalock et al. (2020) also describes gradient-based pruning as the typical (and competitive) alternative to weight pruning. That stream of work is classic and widely known, but it is completely ignored by the authors [7,9-16].

# Comments on improving the paper

I could not understand the necessity and relevance of quivers for what the authors did. I am not saying that there is not one, and in fact deep learning has benefited a lot from the many approaches brought in from other disciplines, but I believe that the paper would benefit immensely from an even smoother introduction of the topics that are not commonly used in our literature. To mention one example of that being done in a masterful way, I would recommend that the authors look at how [17] describes tropical algebra before applying it to neural network.

In Example 5.1, for the shifted ReLU as h(r) = ReLU(r-b) for b positive, I believe you should be explicit about r being nonnegative. By checking the reference cited, I actually found out the definition of a Norm-ReLU as h(|f(x)|) = ReLU(|f(x)|-b) for b positive, which is actually easier to understand. 

I also missed some figures illustrating what is it that makes an activation useful for your approach. I spent some time playing around with your definitions to make sense of it, which is how I got some intuition for the results that you obtained and why they should be valid. Please consider including examples like this one:

https://www.wolframalpha.com/input/?i=plot+%7C%7Cx%7C+-+5%7C

While playing with it, I also started wondering that perhaps there is no need to require b to be positive in the case of what you call a shifted ReLU. Is it not true that your approach would work with the function below?

https://www.wolframalpha.com/input/?i=plot+%7C%7Cx%7C+%2B1%7C

# References cited

[1] https://arxiv.org/abs/1711.02114

[2] https://arxiv.org/abs/2001.00218

[3] https://arxiv.org/abs/2007.06567

[4] https://arxiv.org/abs/2102.07804

[5] https://arxiv.org/abs/1810.02340

[6] https://arxiv.org/abs/1906.06307

[7] https://arxiv.org/abs/2002.07376

[8] https://arxiv.org/abs/2006.05467

[9] https://proceedings.neurips.cc/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf

[10] https://proceedings.neurips.cc/paper/1992/file/303ed4c69846ab36c2904d3ba8573050-Paper.pdf

[11] https://arxiv.org/abs/1506.02515

[12] https://arxiv.org/abs/1611.06440

[13] https://arxiv.org/abs/1705.07565

[14] https://arxiv.org/abs/1810.02340

[15] https://arxiv.org/abs/1905.05934

[16] https://arxiv.org/abs/2004.14340

[17] https://arxiv.org/abs/1805.07091
summary_of_the_review: # Final comments

There are two items affecting my score at this point:

1) How applicable is this result? In other words, how often would we consider using radial neural networks in practice, or how much of this paper could be applied in a broader sense?

2) It is not easy to follow this paper when it gets more technical. I know that many reviewers would not be willing to reassess their score if the authors put the effort of rewriting the paper considerably, but you have my word that I would reread the paper and reconsider my assessment in case you do that.
review_title: null

Meta-review: Authors developed a reparameterization scheme using QR decomposition to reveal symmetries in networks with radial activation. While I welcome new ideas and formalisms from other fields, the ideas presented in this manuscript are fairly straightforward under the radial activation assumption. Although the authors claim that the results may generalize, no evidence was provided. The practical contributions are marginal and lacks comparisons with related DNN compression schemes. Through the review process, the paper has been greatly improved, but unfortunately, this interesting paper does not meet ICLR's standard as is.
Decision: Reject