Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new notion of layer sparsity for neural networks that aims at simplifying network architectures and reducing the number of parameters. A type of regularizers has been introduced to encourage this specific structure.

Overall the paper is well written and the idea is interesting. It pushes the network towards less nonlinear layers if not needed. Does the method apply to scenarios where there are bias terms between the layers?

The numerical results are not sound. Using a fixed number of epochs in each setting, we have no idea of the convergence status for different methods. The authors claim in the end of the results section that all methods can sometimes provide accurate prediction if the number of epochs is extremely large. Does this happen to the numerical examples presented here? Plotting some training curves might help readers understand more of the behavior.

It is fairly standard practice to tune the regularization parameters based on cross-validation or an independent validation set. The authors should include this for the completion of the manuscript, even if the numerical experiment might serve as a proof of concept.

It would be better to include comparison with other sparsity-inducing methods such as connection sparsity and node sparsity. Also, it would be interesting to see the performance in other settings than the most favorable one, for example on some real data sets.
review_title: Major Revision Needed

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:
 
The paper proposes a regularizer enforcing a novel form of sparsity that authors call "layer sparsity". Under certain conditions on layer weights, two consecutive layers in a deep neural network (with certain nonlinear activation functions) can be represented exactly as a single layer. The authors proposed a regularizer that can lead to such layer collapse thus resulting in shallower and more compact models.


##########################################################################

Reasons for score: 

Overall, I vote for rejecting this paper. In my opinion, the proposed regularizer is quite simple and intuitive, but is at the same time novel and could potentially be useful in practice. However, the proposed empirical studies shed very little light on how effective this method can be in practical and even remotely realistic circumstances. I am afraid that the current empirical evaluation is not sufficient.


##########################################################################

Pros: 

1. The paper is clearly written. The core idea of the layer sparsity regularizer is introduced and explained with a sufficient level of mathematical rigor.

2. The proposed technique could become a useful addition to the deep learning practitioner's toolbox.

 
##########################################################################

Cons: 

1. While the experiments described in the paper are a reasonable first step in exploring the effect of adding the proposed regularizer, more empirical exploration (particularly with realistic and practical models) might be necessary for a sufficiently thorough analysis. Currently there is just not enough information to judge the effectiveness of this method in any practical circumstances.

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above. (I will update the score depending on the authors reply.)

I also have a question (and doubts) about Theorem 1 and its proof in Appendix A.

To illustrate my confusion consider a simple model with ReLU activations $f(x)$.
Then $o_i = f\left(\sum_j w_{ij} f\left(\sum_k w_{jk} x_k\right)\right)$ can be rewritten as $o_i = \sum_j f\left(\sum_k w_{ij} w_{jk} x_k\right)$ assuming that all $w_{ij}\ge 0$.
Even after reading Appendix A, I do not understand how dependence on index $j$ inside $f(\cdot)$ can be factored out.
In other words, I do not understand how this expression can be written as $f^{j,j+1}(V^{j,j+1}z)$ with $V^{j,j+1}\in \mathbb{R}^{p_j \times p_{j+2}}$ and $f^{j,j+1}:\mathbb{R}^{p_j}\to \mathbb{R}^{p_j}$.
I think I can even come up with simple numerical examples illustrating this point (all we need to do is make sure that the argument of $f$ is negative for one particular $j$ thus making this term vanish completely in the original expression, but still be present in the result of Theorem 1).

In a long derivation in Appendix A, in a transition from line 2 to line 3, the expression with an index $m$ fixed externally is seemingly replaced with a scalar product involving a summation over this index (at which point it disappears altogether from inside the activation function). It is entirely possible that I do not understand something trivial, but I would ask the authors to explain this transition.

#########################################################################

Other minor typos:

1. A. Barron and J. Klusowski 2018 reference is repeated twice (there must be two records for it in the bibliography file).

#########################################################################

Post-rebuttal.

I would like to thank the authors for their reply, which addressed some of my questions. While I now agree with the main theoretical result when applied to a ReLU nonlinearity, this of course also reduces the area of applicability of the proposed technique. I am also happy to see additional empirical results, which I believe will be the key to making this paper much stronger (since the current theoretical result is actually quite straightforward when applied to the ReLU nonlinearity). But I think the results are still a bit insufficient to make this submission sufficiently strong. One of my concerns is the final accuracy in Figure 2. We can see that the unregularized model surpasses the accuracy of the refitted model and could potentially get higher (or even much higher) should it not have been cut at <100 epochs. I will be excited to see an updated and improved version of this paper in the future, but in my opinion, the current version still needs a bit of work and is not entirely convincing.
review_title: Interesting method, but insufficient empirical analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: __how I would summarize the paper.__
The paper gives an interesting new paradigm of the neural network compression called the *layer sparsity*. The paper builds on the (somewhat underappreciated) observation of Barron&Klusowski that positive weight parameters of multiple layers can be aggregated and reparametrized through the positively homogeneous activation functions. Based on the observation, the paper designs a regularizer that enhances the possibility of such aggregation by regularizing the negative parts only. Whenever successfully regularized, the number of layers in the model can be reduced without sacrificing too much performance, as empirically verified; actually, using the regularizer per se helps reducing the test mean-squared error.

__review tl;dr.__
While the idea itself is quite interesting, I believe that there should be more practical evidence and algorithmic extension of the framework, for the idea to fully bloom.

__what I like.__
The underlying idea itself is quite interesting (somewhat reminiscent of the [lookahead pruning](https://openreview.net/forum?id=ryl3ygHYDB)), and the design of the regularizer makes a perfect sense. Also, potential practical impact of the notion of layer sparsity seems to be big, as reducing the number of layers has a more straightforward practical benefit, in terms of reducing the inference flops/time.

__what I think is missing.__
Model compression became an important research area after deep, convolutional networks gained popularity. Perhaps this is why most of the works on network pruning focus on evaluating their algorithms on compressing deep convolutional networks trained on real (as opposed to synthetic) datasets. To fully persuade the benefit of the proposed algorithm, I think there should be (1) an extension of the algorithm to the aggregation of convolutional layers, which may lead to a larger kernel widths, which is okay, (2) an empirical validation on larger models trained on non-synthetic datasets.

A minor concern is that the refitting step does not seem particularly novel, as the retraining step is already typical in model compression literature.

__a question.__
Would it be possible to compress the layer which does not perfectly satisfy the criterion (with a slight abuse of terminology) at "page 6, penultimate line," but only approximately 0?
review_title: Interesting idea, but maybe that's not enough?

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper introduces a new, interesting definition of "sparsity" in a deep network. It penalizes as a single group the positive values in a layer. When combined with an activation function such as ReLU, they show that this allows to remove entire layers by merging two adjacent weight matrices.

The paper is well written and relatively easy to follow. The mathematical notation could be simplified (especially the difference between W and V).

However, the experimental results are definitely below what one expects in a deep learning paper today. The paper only considers a very simple, artificial setup with "Datasets of 150 samples" and tuning parameters "calibrated very roughly by hand". In my opinion this is not sufficient to show that a method is useful in a realistic context.

The paper should provide some discussion of the relation with recent literature on the lottery ticket hypothesis, especially when introducing its retraining procedure.
review_title: Good ideas with poor experimental evaluation

Meta-review: Reviewers agree that the idea of layer wise regularization is interesting and is in line with many efforts in the optimization realm to specialize in the training procedure and the learning rate to each layer.  Given the depth of some state of the art neural networks, efficiency is at stake and the idea brought up in this paper naturally falls into that.  While the theoretical result in Theorem1 is sound and clear, an extended result on the impact of such « merge » and « layer skipping » on the overall predictions of the algorithm can be well appreciated. The overall goal of network compression should remain to reduce drastically the network size, and thus the training time (energy consumption etc...), while keeping a relatively good prediction accuracy (at least of the same order). Being able to back this with theory (and of course experiments) is crucial.   Reviewers also pointed out that the empirical evaluations were not sufficient for ICLR. For example, there are no enough comparisons with existing algorithms and there should be more experimental results based on real datasets. Although the rebuttals did help clarify some of the issues raised by the reviewers, overall this paper does not seem to meet the bar to be accepted. 


Decision: Reject