Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1. Summary
The authors of the paper compare the learning of representations in DNNs with Shannons channel coding theory, which deals with reliably sending information through channels. In channel coding theory the statistical properties of the coding of the information can be designed to fit the task at hand. With DNNs the representations cannot be designed in the same way. But the representations, learned by DNNs, can be affected indirectly by applying regularization. Regularizers can be designed to affect statistical properties of the representations, such as sparsity, variance, or covariance. The paper extends the regularizers to perform per-class regularization. This makes sense, because, for example, forcing the variance of a representation to go towards zero is undesirable as it would state that the unit always has the same output no matter the input. On the other hand having zero variance for a class is desirable as it means that the unit has a consistent activation for all samples of the same class. The paper compares different regularization techniques regarding their error performance. They find that applying representation regularization outperforms classical approaches such as L1 and L2 weight regularization. They also find, that performing representation regularization on the last layer achieves the best performance. Class-wise methods generally outperform methods that apply regularization on all classes.

2. Remarks
Shannons channel coding theory was used by the authors to derive regularizers, that manipulate certain statistical properties of representations learned by DNNs. In the reviewers opinion, there is no theoretical connection between DNNs and channel theory. For one, DNNs are no channels in the sense that they transmit information. DNNs are rather pipes that transform information from one domain to another, where representations are learned as an intermediate model as the information is being transformed. Noise introduced in the process is not due to a faulty channel but due to the quality of the learned representations themselves. The paper falls short in explaining how DNNs and Shannons channel coding theory fit together theoretically and how they used it to derive the proposed regularizers. Despite the theoretical gap between the two was not properly bridged by the authors, channel coding theory is still a good metaphor for what they were trying to achieve.
The authors recognize that there is similar research being done independently by Belharbi et al. (2017). The similarities and differences between the proposed work and Belharbi et al. should be discussed in more detail.
The authors conclude that it is unclear which statistical properties of representations are generally helpful when being strengthened. It would be nice if they had derived at least a set of rules of thumb. Especially because none of the regularizers described in the paper only target one specific statistical property but multiple. One good example that was provided, is that L1-rep consistently failed to train on CIFAR-100, because too much sparsity can hurt performance, when having many different classes (100 in this case). These kinds of conclusions will make it easier to transfer the presented theory into practice.

3. Conclusion
The comparison between DNNs and Shannons channel coding theory stands on shaky ground. The proposed regularizes are rather simple, but perform well in the experiments. The effect of each regularizer on the statistical properties of the representation and the relations to previous work (especially Belharbi et al. (2017)) should be discussed in more detail. 
review_title: Paper proposed new regulariziation schemes

final_score: rating: 5: Marginally below acceptance threshold
review_content: This is a well-written paper which starts on a good premise: DNNs learn representations; good representations are good for prediction; they can be seen as codes of the input information, so let us look at coding/communication/information theory for inspiration. This is fine and recent work from N Tishby's group develop some intriguing observations from this. But this paper doesn't follow through the information/communication story in any persuasive way. All that is derived is that it may be a good idea to penalise large variations in the representation -- within-class variations, in particular. The paper does a good job of setting up and comparing empirical performance of various regularizers (penalties on weights and penalties on hidden unit representations) and compares results against a baseline. Error rates (on MNIST, for example) are very small (baseline 3% versus the best in this paper 2.43%), but, if I am right, these are results quoted on a single test set. The uncertainties are over different runs of the algorithm and not over different partitions of the data into training and test sets. I find this worrying -- is there a case (in these datasets that have been around for so long and so widely tested), there is a commmunity-wide hill climbing on the test set -- reporting results that just happen to be better than a previous attempt on the specific test set? Is it not time to pool all the data and do cross validation (by training and testing on different partitions) so that we can evaluate the uncertainty in these results more accurarately?
review_title: DNN Representations as Codewords: Manipulating Statistical Properties via Penalty Regularization

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a set of regularizers which aims for manipulating the statistical properties like sparsity, variance and covariance. While some of the proposed regularizers are applied to weights, most are applied to hidden representations of neural networks. Class-wise regularizations are also investigated for the purpose of fine-grained control of statistics within each class. Experiments over MNIST, CIFAR10 and CIFAR100 demonstrate the usefulness of this technique.

The following related work also studied the regularizations on hidden representations which are motivated from clustering perspective and share some similarities with the proposed one. It would be great to discuss the relationship.

Liao, R., Schwing, A., Zemel, R. and Urtasun, R., 2016. Learning deep parsimonious representations. NIPS.

Pros:
(1) The paper is clearly written.

(2) The visualizations of hidden activations are very helpful in understanding the effect of different regularizers.

(3) The proposed regularizations are simple and computationally efficient.

Cons:
(1) The novelty of the paper is limited as most of the proposed regularizers are more or less straightforward modifications over DeCov.

(2) When we manipulate the statistics of representations we aim for something, like improving generalization, interpretability. But as pointed out by authors, improvement of generalization performance is not the main focus. I also do not find significant improvement from all experiments. Then the question is what is the main benefit of manipulating various statistics? 

I have an additional question as below:
In measuring the ratio of dead units, I notice authors using the criterion of “not activated on all classes”. However, do you check this criterion over the whole epoch or just some mini-batches?

Overall, I think the paper is technically sound. But the novelty and significance are a bit unsatisfactory. I would like to hear authors’ feedback on the issues I raised.

review_title: This paper presents a set of regularizers which aims for manipulating the statistical properties like sparsity, variance and covariance.

Meta-review: The paper received scores of 5,5,5, with the reviewers agreeing the paper was marginally below the acceptance threshold. The main issue, raised by both R2 and R3 was that connection between representation learning in deep nets and coding theory was not fully justified/made.  With no reviewer advocating acceptance, it is not possible to accept the paper unfortunately. 
Decision: Reject