Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: The paper focuses on defining a new architecture that allows being reduced without significantly affecting the performance. 

In short, the paper is not properly written nor well organized; is hard to read with vague contributions and vague positioning with respect to the state of the art. Experiments are not convincing: Toy experiment and minimum experiments in MNIST without a clear comparison to existing neuron pruning algorithms.  
review_title: Not a clear paper, nor contributons, lack of convincing experiments and positioning with respect to state of the art methods. 

final_score: rating: 3: Clear rejection
review_content: Summary: 
- In this paper, the authors propose a novel algorithm for pruning fully-trained ReLU neural networks. To motivate the algorithm, the authors first introduce a new network architecture with an affine skip-connection at each layer. Then the authors connect it to the theory developed in an `unpublished work`. They show that for such neural networks, the number of units in the original layer can be greatly reduced. The main idea of the proposed algorithm is basically to prune those neurons whose removal will not change the function a lot. To quantify this, the authors turn to the L2 norm of each neuron. Experiments on simple toy data and MNIST are conducted.


Overall, I believe this paper is not ready for publication. So, I vote for rejection. 
- This paper massively refers to the unpublished work by the author, while the authors only provide only little details about the developed theory in the unpublished work. If this is a concurrent submission to ICLR, the author should still cite it anonymously.
- To me, the proposed deep stack network is very similar to the formulation of the highway network in Srivastava et al., (2015).
- The experiments are not convincing enough. The authors only conduct experiments on a simple toy dataset and MNIST. The numbers are fairly close to each other. To show the statistical significance, some measures, such as one standard error should be provided. I would encourage the authors to at least conduct some experiments on CIFAR datasets.

Typos:
- Exactly speaking, the learned function will not be the same same as before ->Exactly speaking, the learned function will not be the same  as before 
- Therefore the the function optimizing eq. (2) can be represented by finite number -> Therefore the function optimizing eq. (2) can be represented by a finite number


Srivastava, Rupesh Kumar, Klaus Greff, and Jürgen Schmidhuber. "Highway networks." arXiv preprint arXiv:1505.00387 (2015).
review_title: Paper is not ready for publication.

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The authors leverage a functional regularization reformulation of neural network training problems to prune networks via a reduction algorithm. They present limited experimental evidence showing that the reduction algorithm reduces the number of neurons without sacrificing too much accuracy. 


Major comments/questions

1. Clarity and correctness
There are significant issues in the presentation and clarity. The authors use footnotes to explain important concepts, but many definitions are missing. The material in the footnotes can be included in the main text with a more natural flow. The main observation in Section 4.1 is not presented as a rigorous result, which appears to be the most interesting result. Remark 4.1 can also be presented as a theorem.

2. Insufficient comparisons with the baselines.
 There is extensive literature in pruning and sparsification of network layers. In Table 1 and Table 2 there is no comparison with standard baselines in pruning. Does the proposed method perform better than standard pruning based on  weight magnitude/gradient norm/Hessian based metrics? 

3. Figure 6 is not very informative. It would be better to zoom in the relevant portion of the plot. Focusing on a few examples instead of seven different examples would make a better display.

4. It is not clear why the authors consider the bottleneck architecture in Figure 4 and 5. Is the bottleneck required for the theory behind pruning or reducing overfitting?

5. Section 3 starts with 'the authors have shown in an unpublished paper". Is this referring to Maennel et al, 2018? Similar results also appeared in other papers (e.g. Savarese et al. 2019. Please provide a reference or proof for the equivalence of (1) and (2). The proof is in fact straightforward, we only need n_j>=d+1 to hold for Caratheodory's theorem. The authors can be more precise for the required width n_j. This is a significant omission.

6. On page 3, footnote 1, P(f) is not properly defined and it's not clear what P(f)=\infty means. This can be clarified by providing necessary references noted above.
Furthermore in eq (1), NN_\theta is not properly defined. Is this a standard relu network?  

7. In the introduction, the authors claim that the proposed method preserves the network output exactly as opposed to other pruning methods. However, in Section 4.2 and 4.3, the authors also resort to approximation methods involving magnitude based pruning and clustering, which are standard in the literature. The observation from Section 4.1 is also not exactly applied and yields an approximate neural network.

8. What would be the computational complexity of looping through every neuron and the proposed approximation? Is there a way to justify this approximation?

Minor comments
1. The manuscript needs a careful proofreading since it contains lots of typos and grammatical errors.
page 1. It's architecture -> Its architecture.
page 1. authros -> authors

2. There some definitions which need further explanation
page 1. I believe what is meant by a 'large layer' is a wide layer.
page 1. Could you please clarify what sparsity refers to in "reduce the number of neurons by 90% to 99% without introducing sparsity"? 
page 3. bottleneacks->bottlenecks
review_title: This paper considers a functional regularization form of neural network training problems to prune networks. There are significant issues in the presentation and clarity.

final_score: rating: 2: Strong rejection
review_content: The paper suggests a pruning technique specific to ReLU networks by taking advantage of activation patterns and the separating hyperplanes. The technique consists of three steps: : (1) remove neurons that are never active and combine neurons that are always active, (2) remove neurons with little contribution to the output, and (3) use weighted k-means to combine other neurons. The method is evaluated on specific toy problems and the MNIST dataset.

The paper proposes a novel technique to reduce neurons in a ReLU network. The method to combine neurons takes the hyperplane arrangements (where activations of neurons change) into account and leads to much smaller networks with equal performance in the experiments.

The three major problems of the paper are that it lacks motivation of the proposed technique, it contains an insufficient experimental evaluation and important parts of the paper cannot be reviewed either due to a reference to unspecified, unpublished work or a lack of a derivation.

Since the correctness of the paper cannot be evaluated and the technique is insufficiently validated, I recommend to reject. 


Details on weaknesses:
- The correctness of the proposed technique cannot be reviewed. 
(a) Section 3 cannot be confirmed as it refers to results from unspecified unpublished work, i.e., it is impossible to find and read through the unpublished work to estimate its validity. Moreover it is unclear why these results are important for the given paper and how the results are used. The paper states that one should only learn from this entire section that a finite(!) number of neurons is sufficient (which we always have in a practical setting so the conclusion is void?) In any way, either this section is not necessary for the rest of the paper and should be removed, or it is necessary in which case it cannot be verified.
(b) The proposed pruning technique consists of three steps (see above in the summary) two of which are trivial: (1)&(2). The third step (3) uses weighted k-means to combine other neurons. There is no explanation, motivation or derivation of the equations how the clusters are combined. (The workings of the method are also surprising, because a cluster containing a single neuron is reduced to a new single neuron in such a way that the function changes, which is counter-intuitive. It seems therefore likely that the equations contain typos, also since they introduce square roots of square roots and it is not defined what is meant by squaring a vector in the function g.)
Therefore, the validity cannot be confirmed and the reader must trust the experimental result section. This is unfortunate as there are opportunities to shorten less relevant parts in favor of a derivaiton of the equations.

- The experiments are not sufficient. The experiments only consider specific toy problems and the MNIST dataset, which is too simple to showcase the pruning technique. The method is neither compared to any other pruning technique and it is fairly simple to prune networks on MNIST with a similar loss of accuracy. Finally, the method introduces two hyperparameters which are not tested. To validate the performance, both a more complex dataset and the comparison to other pruning techniques are necessary.

- The presentation needs improvement. For example, there is a rather long explanation of a seemingly simple architecture and still details are left unclear (Is there a linear layer with weights that are trained, or is the linear skip connection only introduced when pruning the network?) It would be helpful to add an equation for the stack layer and reduce the explanations. The possibly redundant Section 3 could be removed. Instead, the proposed method could be derived and explained. The experiments could be explained in more detail (Why do the plots show the smoothened second derivative?)




Typos:
Line 4 Motivation „authros“
Page 3, footnote, „bottleneacks has diomension“
Page 4 toward the end of Section 3: „the the“
Page 5: the first sentence is not a sentence
review_title: Pruning technique for ReLU networks with insufficient validation and derivation

final_score: rating: 2: Strong rejection
review_content: The paper describes a way to reduce the dimensionality of a deep ReLU network. In general, the paper is not well written and hard to follow. They keep referencing "unpublished work of the authros" although its use in practice is not very clear.

Practically, they prune a deep network by (a) removing dead ReLU neurons; (b) combine neurons for which the ReLU always acts as the identity. None of these two ideas is particularly novel, especially considering the huge amount of literature to be found on network pruning. Experiments are only done on an artificial dataset and on MNIST.

Many parts of the paper are poorly described. For example, their "stack network" is simply a residual network with affine projections on the residual link. A "P-FUNCTIONAL" is not defined. The link between Eq. (4) and their algorithm is not clear.
review_title: Not novel, below average on most aspects

Meta-review: This is a clear reject. None of the reviewers supports publication of this work. The concerns of the reviewers are largely valid.
Decision: Reject