Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In their submission the authors discuss an alternative learning rule to backpropagation called “maximal coding rate reduction” that was introduced recently in Yu et a. 2020. As far as I could tell, as a non-expert, maximizing the coding rate reduction objective encourages inputs from different classes to be maximally incoherent with respect to one another (spanning the largest possible volume) while inputs from the same class should be highly correlated.

In the present submission, the authors show that gradient ascent on the coding rate reduction naturally takes the form of a “neural network” with a particular residual network architecture that they term ReduNet. The authors argue that sending an input through the layers of ReduNet naturally leads to representations that (approximately) maximize the coding rate. The authors continue to show that when the representation is required to be group-equivariant with respect to shifts, a convolutional structure naturally emerges. This allows the authors to compute the update using layerwise fourier transformations. Finally, the authors show that on several example tasks ReduNet leads to outputs that have a large coding rate with no parameters learned via backpropagation.

There were several aspects of this submission that I liked a lot. I think the construction seems interesting and the rate reduction metric seems like a reasonable thing to optimize. I found the relationship of coding rate maximization to ReduNet to be quite clever and the experiments seemed to suggest that the approximations employed in the paper (e.g. the estimated membership in eqn 9) did not spoil the mapping.

Having said this, I don’t know that I agree with the authors’ interpretation of their results and I believe they might be somewhat overstated -- of course I am happy to be corrected if I am incorrect in my assessment.

1) My primary concern is the claim that the maximum rate reduction principle gives first-principles insight into convolutional networks. The authors show that a particular convolutional architecture, namely ReduNet, approximates projected gradient ascent on the MCR2 metric. However, it is not at all clear to me that standard convolutional networks or residual network architectures are well-described by ReduNet. In fact, it seems that the very particular structure of ReduNet has a number of features that are at odds with standard convolutional networks: nonlinearity only enters into ReduNet via the approximate membership (which enters via a softmax-like function) and outputs of each layer are projected onto the (n-1)-sphere. 

2) Since the relationship between ReduNet and commonplace CNNs seems tenuous to me, it seems there is a significant burden on the authors to probe this question empirically. For example, do CNNs trained via backpropagation lead to architectures that resemble ReduNet? Unfortunately, as far as I could tell, the authors did not include experiments of this type.

3) If the authors are proposing ReduNet as an alternative to modern CNNs, it would be nice to see how ReduNet performs on some common tasks with well-established baselines (possibly with some fine-tuning step). However, as far as I could tell most of the experiments focused on verifying the properties of ReduNet. Note that I do not think one needs to achieve state-of-the-art performance here, but it would be nice to see how ReduNet fairs since this will probably affect its impact.

A few more minor points:

1) I may be missing something, but given the construction of ReduNet, I feel as though the emergence of a convolutional structure subject to translation invariance is not terribly surprising. Indeed I feel like this result has high overlap with Cohen and Welling [1]. This is not necessarily a criticism of the result per-se, but I think the phrasing could do a bit more to note this connection.

2) It was not obvious to me why Z had to be constrained to S^{n-1}. Can the authors provide some intuition here?

3) The authors note that non-linearity enters the network via the estimated membership. However, as far as I could tell, the estimated membership is only used for test points. Does this imply that the network is linear when the membership is known?

[1] Cohen, T. and Welling M. “Group Equivariant Convolutional Networks” JMLR 2016

Update: After discussion with the authors, I am inclined to lower my score. While I find the architecture proposed by the authors to be interesting, I do not think they have done enough to motivate the connection with neural networks. I find this especially troubling since the language used by the authors continues to imply that the connection is obvious. I would encourage the authors to look into the literature on scattering networks (e.g. https://arxiv.org/abs/1203.1513) for another approach to explaining networks from first principles that, I think, does a better job of making the connection to realistic neural network architectures. 

review_title: An interesting paper that may leave several outstanding questions.

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Summary
This paper proposes a theoretical understanding of neural architecture using the  principle of rate reduction. Yu et.al 2020 and the derived optimization steps naturally leads to operations such as network layers and identity residual. By enforcing shift invariant, it can also lead to convolutional operation. The network can be constructed with a forward propagation fasion, which conserves good discriminative ability. 

#### Novelty
Theorectical guidance of network design is one of the key direction in representation learning. 
The paper proposes a novel perspective (rate reduction) in network construction that generalized to the design of networks such as resnet and resnext. However, there are a bunch or other networks such as denseNet, non-local network etc., which are also performing well in practice and designed with huerestics of larger context and better gradient propagation, will the author also include these representations within the objective of rate reduction? 

Or does the generation process from rate reduction (compact discriminative representation) come with additional guidance of what network, or answer the question in introduction, what is the object is optimal for network design, which could show to be effective on multiple tasks. 

I think the major issue of current result is that we can see the objective explain  that the set up of networks (resNet) seeks a compact representation, but the author has not shown strong experiments that their yielded alternatives by optimizing the objective (rate reduction) has positive relation with the network generalization. Is higher rate reduction leads network with better performance on multiple tasks ? 

Last, it is obvious that shift invariance leads to convolution, which is effective for classification of 2D objects. However, in realistic senario, we may seem equivalent disentangled representation rather  than invariance, is it possible for the objective leads to convolution without explicit inducing shift invariance ?  

#### Writing and reference

The writing is good and easy to follow, checked few derivatives which are correct. I think the paper could also related to optimization inspired network design, e.g. Optimization Algorithm Inspired Deep Neural Network Structure Design, which delivers another perspective. 


In general, I think the objective is novel, but not generalized enough to explain lots of high performance networks yet. However more exploration of theoretical study should be encouraged.  

review_title: Novel perspective for deriving network architecture. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a network derived from maximizing of rate reduction, known as the MCR^2 principle, where  all parameters are explicitly constructed layer by layer in a forward propagation fashion. Compared with exiting work by Yu et al. (2020), the proposed work is more like a "white box" that each layer is more interpretable.  Results showed the proposed network can learn a good discriminative deep representation without any back propagation training. The derivation of the network also suggests that the network is more efficient to learn and construct in the spectral domain.

Overall, the proposed work looks reasonable. The paper is well-structured. The derivation and experiments seem convincing.  Unfortunately, the proposed work is out of the reviewer's expertise and therefore it is hard to provide valuable comments.
review_title: interesting paper with solid analysis

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors propose a deep network approach using the principle of rate reduction as a loss under a gradient ascent approach, avoiding traditional backpropagation. Besides, the work attempts to interpret the proposed framework from both geometrical and statistical views. Then, shift-invariant properties are discussed. The innovative method allows the inclusion of a new layer structure named ReduNet, which could benefit the deep learning community. Though the experiments are not challenging concerning the studied databases, the authors aim to probe the concept without a complete implementation tuning. Overall, the paper is illustrative enough regarding the mathematical foundation.
review_title: The paper is well-founded and presented with an innovative deep network perspective

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper formulates an iterative process of deriving encoding of data into feature space as a deep model, called ReduNet, where each layer corresponds to iteration of the optimisation process the feature space according to the MCR^2 principle.  The MCR^2 optimisation maps points of different classes into separate subspaces, with volume of each subspace being minimized while the volume of the entire space is maximized.   It is analogous to pushing like things together and unlike things apart.  The novelty of the paper is in that formulation of the feature optimisation is baked-in into a deep architecture. 

MCR^2 principle seems like a sensible approach to learning, especially given that embedding algorithms (such as face encoding) use it already.  It’s nice to see some rigorous mathematical treatment on this.  However, I get confused pretty early on by the notations.  If f(x,\theta)\in \mathcal{R}^k and z=f(x)\in \mathcal{R}^n….then since y=h(z), then f(x,\theta)=h(f(x))…and so f(x,\delta) and f(x) are two different functions.  Yet later in the text z=f(x,\theta).  And from then on, including equation 11, f(x,\delta)=\psi^L(z_L+\etag(z_{L-1},\theta_{L-1})…which would make it seem f(x,\theta)\in \mathca{R}^n.  And what is g(z_l,\theta_1)?  Equation 8 tells us what g(z_l,\theta_1) must approximate…but what is it exactly?   Is that a neural network, or some model, with parameters \theta_l?  Or is Equation 8 a definition of g(z_l,\theta_l)…in which case what is \theta_l?  I don’t think the math is necessarily wrong…just notation is confusing and definitions changing/not consistent.

I have also questions about equation 11, where number of layers is equivalent to iterations while maximizing MCR^2 and the width of each layer corresponds to m, the training points in the dataset.  So, in order to do a mapping of an input x, we need to perform L iterative steps using the entire m points every time?  Isn’t that equivalent to doing a massive learning process, using the entire dataset, for each mapping?  How computationally costly is that?  I also don’t quite understand how \psi^l(z_1,\theata_1) works - how does \theta_l change over iterations?  Experimental section is not helping me with this, since it’s stated that E, C^j are computed for each layer…but there are no details on what \theta_l is and how g(z_1,\theta_1) is evaluated.  And if f(x,\theta)=z^L…then how do we get classification from that?  Is it just based on definition of \hat{\pi}^j(z_l) from page 4?

Finally, I am not sure if the result of obtaining a convnet architecture in ReduNet when translation invariance constraint is added the embedding is all that surprising.  Isn't it somewhat obvious that if each layer of ReduNet is invariant in some way, then the entire network is invariant?  It feels like that what we are learning here is not that in order to have translation-invariant mapping we must have a convent...but rather that we can obtain a translation invariant deep architecture with translation invariant layers. 
review_title: MCR^2 principle makes sense, details of ReduNet are confusing

Meta-review: This paper received borderline scores, which makes for a difficult recommendation. Unfortunately, two of the reviews were too short and thus were of limited use in forming a recommendation. That includes the high-scoring one, which did not adequately substantiate its score.

There is much to admire in this submission. Reviewers appreciated the originality of this research, linking rate reduction optimization to deep network architectures:
* R1: "The paper proposes a novel perspective"
* R4: "The novelty of the paper is in that formulation of the feature optimisation is baked-in into a deep architecture"
* R5: " I think the construction seems interesting and the rate reduction metric seems like a reasonable thing to optimize. I found the relationship of coding rate maximization to ReduNet to be quite clever"
* R3 (short): "The innovative method allows the inclusion of a new layer structure named ReduNet"

Reviewers also applauded the paper's clarity, including R4 who raised their score to 6 based on satisfying clarity revisions from the authors:
* R1: "The writing is good and easy to follow"
* R4 post-discussion: "Clarity is not an issues anymore - additional explanations provided by the authors and one more careful reading of the paper helped in understanding of all the aspects of the model"
* R2 (short): "The paper is well-structured."

However, there were some core questions around how well the main significance claims of the paper are supported. The most in-depth discussion on these topics is in the detailed thread with R5. In that thread there are many points discussed, but the two issues seem to be:
1. whether the connection between ReduNet and standard neural net architectures is sufficiently substantiated so as to constitute an explanation for behaviors of those standard architectures, like CNNs; and
2. whether the emergence of ReduNet's group invariance/equivariance is surprising or qualitatively new.

The first is much more central. On the first issue, R5 writes in summary:
"Fundamentally I think the authors propose a hypothesis: that ReduNets explain DL models. However, the authors do not take meaningful steps towards validating this hypothesis. [...] I would contrast this with, for example, the scattering networks paper (https://arxiv.org/abs/1203.1513) which did an exceptional job of arguing for an ab initio explanation of convolutional networks."

I find R5's perspective on this point to be compelling, in that the paper currently doesn't do enough to justify these main claims, either through drawing precise nontrivial mathematical connections or through experimental validation. (The thread has a much more detailed and nuanced discussion.)

The second issue is not quite as central to the significance of the paper, but it was noted by multiple reviewers:
* R5: "I may be missing something, but given the construction of ReduNet, I feel as though the emergence of a convolutional structure subject to translation invariance is not terribly surprising."
* R4: "Finally, I am not sure if the result of obtaining a convnet architecture in ReduNet when translation invariance constraint is added the embedding is all that surprising."
* R4 post-discussion: "Reading the exchange between the authors and R5 I am still not fully convinced that translation invariance property is all that surprising, but for me that's not a reason to reject."

At the least, the paper as written hasn't yet convinced some readers (myself included) on these claims.

As I mentioned at the start, this paper is borderline, but because I am largely aligned with R5's perspectives, I think this paper does not quite pass the bar for acceptance. I recommend a rejection, but I look forward to seeing a strengthened version of this work in the future. I hope the feedback here has been useful to bringing about that stronger version.
Decision: Reject