Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes and tests two ideas. (1) a method of pruning networks by identifying highly correlated neuron pairs, pruning one of the pair, and then modifying downstream weights to compensate for the removal (which works well if the removed neurons were highly correlated). (2) a method, dubbed NoiseOut, for increasing neuron correlation by adding auxiliary noise target outputs to the network during training.


The first idea (1) is fairly straightforward, and it is not clear if it has been tried before. It does seem to work.


The second idea (2) is of unclear value and seems to this reviewer that it may merely add a regularizing effect. Comments in this direction:
 - In Fig 4 (right), the constant and Gaussian treatments seem to produce the same effect in both networks, right? And the Binomial effect seems the same as No_Noise. If this is true, can we conclude that the NoiseOut targets are simply serving to regularize the network, that is, to reduce its capacity slightly?
 - To show whether this effect is true, one would need to compare to other methods of reducing the network capacity, for example: by reducing the number of neurons, by applying L2 regularization of various values, or by applying Dropout of various strengths. Fig 7 makes an attempt at this direction, but critically misses several comparison treatments: “Pruned without any regularization”, “Pruned with only L2”, and “Pruned with only DropOut”. Have these experiments been run? Can their results be included and used to produce plots like Fig 5 and Fig 7?

Without these comparisons, it seems impossible to conclude that NoiseOut does anything but provide similar regularization to DropOut or L2.


The combined ideas (1) + (2) DO produce a considerable reduction in parameters, but sadly the experiments and exposition are somewhat too lacking to really understand what is going on. With a little more work the paper could be quite interesting, but as is it should probably not be accepted.


Additional comments:
 - Section 4 states: “In all of these experiments, the only stop criteria is the accuracy decay of the model. We set the threshold for this criteria to match the original accuracy; therefore all the compressed network have the same accuracy as the original network.” Is this accuracy the train accuracy or test accuracy? If train, then test accuracy needs to be shown (how much test performance is lost when pruning?). If test, then this would typically be referred to as “cheating” and so the choice needs to be very clearly stated and then defended.
 - Lowercase rho is used to indicate correlation but this is never actually specified, which is confusing for. Just state once that it indicates correlation.
 - How do these results compare to other pruning methods? No numerical comparison is attempted.
review_title: Interesting results and the start of a good paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
In this paper, the authors introduce NoiseOut, a way to reduce parameters by pruning neurons from a network. 
They do this by identifying pairs of neurons produce the most correlated outputs, and replacing the pair by one neuron, and then appropriately adjusting weights.
This technique relies on neurons having high correlations however, so they introduce an additional output neuron -- a noise output, which results in the network trying to predict the mean of the noise distribution.
As this is a constant, it increases correlation between neurons.
Experiments test this out on MNIST and SVHN

Comments:
This is an interesting suggestion on how to prune neurons, but more experiments (on larger datasets) are probably need to be convincing that this is an approach that is guaranteed to work well. 

Equation (5) seems to be very straightforwards?

It seems like that for larger datasets, more noise outputs might have to be added to ensure higher correlations? Is there a downside to this in terms of the overall accuracy?

The paper is presented clearly, and was definitely interesting to read, so I encourage the authors to continue this line of work.

review_title: Good proof of concept, but more experimental evidence needed

final_score: rating: null
review_content: The proof of the effectiveness of noisy units relies on the absence of biases. Otherwise, a simple solution for the noisy outputs has the bias equal to the expected output value and zero weights. Please clarify if your networks use bias terms, and if they use weight decay (which should remove weights incoming to the noisy units). 


review_title: Do you use bias terms?

final_score: rating: null
review_content: Section 4 states: “In all of these experiments, the only stop criteria is the accuracy decay of the model. We set the threshold for this criteria to match the original accuracy; therefore all the compressed network have the same accuracy as the original network.” Is this accuracy the train accuracy or test accuracy? If train, then test accuracy needs to be shown (how much test performance is lost when pruning?). If test, then this would typically be referred to as “cheating” and so the choice needs to be very clearly stated and then defended.

review_title: Train vs. Test acc?

final_score: rating: 3: Clear rejection
review_content: The paper proposes to prune a neural network by removing neurons whose operation is highly correlated with other neurons. The idea is nice and somewhat novel - most pruning methods concentrate on removal of individual weights, however I haven't done a through research on this topic. However, the experimental and theoretical justification of this method need to be improved before publication:

1. Experiments. The authors do not report accuracy degradation while pruning in the tables, laconically stating that the networks did not degrade. This is not convincing. The only details are given in Figure 5, however this Figure disagrees with Table 2: in the Table, the number of parameters ranges from 40k-600k, while the Figure pictures the range 12k-24k. Unless more details are provided, simply claiming that a network can remove 50% neurons with no number on the degradation of accuracy is not convincing.

2. Theory. The proofs do not match the experimental conditions and make unreasonable assumptions. The proofs show that in the absence of biases a network with a constant output will have two correlated neurons that generate the output offset. However, this is exactly why networks have biases and doesn't explain why noise injection helps (the proof suggests that all should be fine with deterministic auxiliary neuron). My interpretation is that the noisy output injects gradient noise (see e.g. the concurrent ICLR submission https://openreview.net/forum?id=rkjZ2Pcxe). As such the proof muddies the picture more than it helps in understanding what is happening.

Verdict:
Reject and resubmit. 
The pruning idea has potential, however its efficiency must be more soundly demonstrated (please provide network accuracies at various pruning levels, the method removes one neuron at a time, this allows making of nice plots) rather than laconically stating that a degradation on mnist from 97% accuracy to 92% is not significant (Figure 5.). Please provide Figures and Tables that agree with the text in terms of numbers provided.
review_title: Intriguing idea, but theory and experiments are lacking.

Meta-review: all reviewers agree that the paper is not convincing enough at this stage but needs more work to be ready for ICLR (e.g. missing comparisons to other existing methods).
Decision: Reject