Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a technique to encourage the within-layer
activation diversity and therefore improve the model performance.
Specifically, they design a within-layer loss that add penalty to the similar
neurons. They also showed that encouraging the within-layer diversity will
help reduce the generalization error. 

The paper is well-presented and authors provided enough intuition as well as
theoretical evidence why the diversity would help. Although I did not check
all the proofs, the results seem to be right. 

The definition of within-layer diversity seems to be simply the
concentration of the values of each individual neuron. How does that affect
the distribution of the layer output on the unit ball? Will this lead to a
output similar to 'binarized' output?

The experiment seems insufficient to support the argument. Only very simple
neural networks on two toy examples are provided. More ablation study of the
neural/layer output distribution would help better understanding this issue.

Overall I think this paper provides some insights to how the generalization
error is related to the neuron outputs and vote for accept.
review_title: Well written paper, could use more ablation study

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes adding regularization terms to encourage diversity of the layer outputs in order to improve the generalization performance. The proposed idea is an extension of Cogswell's work with different regularization terms. In addition, the authors performed detailed generalization analysis based on the Rademacher complexity. The appearance of the term related to the layer output diversity in the generalization bound provides theoretical support for the proposed idea.

The main weakness of this paper, in my humble opinion, is the lack of important details or rigor in the experiments presented. For example, the authors didn't mention how the hyperparameter selection was conducted, what optimizer (and its parameters) was used, how many runs per result and the confidence interval, whether any test was done to establish statistical significance, why state-of-the-art architecture was not used for the image classification tasks, etc. Without these important details and rigorous comparison, it's hard to have high confidence in the reproducibility of the results.

Details:
1) Intro section. The line of work in "double descent" shows that overparameterization doesn't necessarily lead to overfitting. For completeness, it'll be good to mention this line of work and qualify the claim on overfitting.
2) End of section 2. The authors claim that the proposed diversity term induces "within-layer" feedback. The regularization term is computed on the outputs of a layer, which do depend on the parameters of the lower layers. So when backpropagation happens, it will affect the parameters of the lower layers. Therefore, "within-layer" feedback doesn't sound accurate to me.
3) Section 3.1, last bullet point. Should $\tau$ be introduced here?  Otherwise, where does the $\tau$ later used in Lemma 3.5, Lemma 3.6 and Theorem 3.7 come from?
4) Section 5. The proposed regularization terms don't seem cheap to compute for large networks with wide layers. It'll be helpful to measure the training cost increase.
review_title: A neat extension encouraging layer output diversity with theoretical backing

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed three ways of diversifying outputs of neurons, and the analysis showed that the generalisation bound becomes tighter when the neurons become more diversified. It is an interesting finding, along with theoretical results and empirical results. Although, from a practical perspective, there are still many concerns.

It is clear that by increasing d_min, the generalisation bound gets tighter. However, it is also obvious that there are other factors that one can control to make the bound tighter, and regularising other factors might be simpler in terms of implementation and optimisation. 

1. The constant C_4 in the upper bound of the weight vector connecting the hidden-layer to the output neuron. \sqrt(J) decays linearly with C_4, and the first term in the generalisation bound for regression tasks decays quadratically w.r.t. C_4. Compared with a linear decay w.r.t. d_min, C_4 seems to be a better option to regularise neural networks. In practice, one can empose an \ell_2 regularisation on the top linear layer to control the overall norm of the weight matrix so that C_4 is controlled. 

2. The constant C_5 = L_\phi C_1 C_3 + \phi(0). As we can see in the generalisation bound for regression tasks, \sqrt(J) decays quadratically w.r.t. C_5, which is even faster than the decay rate w.r.t. C_4. To control C_5, one can choose an activation function that has a small L_\phi, or to control the weight vectors to the activation function to have a small norm C_3. Both of them can be done relatively easily compared to optimising pair-wise similarity. 

Overall, I think there are other regularisations suggested by the bound that could be put into practice, which might also lead to good generalisation, and also simpler optimisation problem. 
review_title: Missing many potential comparison partners

final_score: rating: 3: Clear rejection
review_content: Strong point: the paper addresses an important problem.

Three main weaknesses, which justify the score:
•	The theoretical developments presented in the paper build on the Rademacher complexity, but ignore the conclusions drawn by Zhang et al. in Section 2.2 of their ICLR 2017 paper (Understanding deep learning requires rethinking generalization).
•	The theoretical developments build on the assumption that (i) there exists a lower bound, valid for any input, to the distance between the output of each pair of neurons, and (ii) the proposed diversity loss increases this lower bound. Those two assumptions are central to the theoretical developments, but are quite arguable. For example, a pair of neuron that is not activated by a sample, which is quite common, leads to a zero lower bound.
•	Experimental validation are not convincing. Only shallow networks are considered (2 or 3 layers), and the optimization strategy, including the grid search strategy for hyperparameters selection, is not described.

Minor issue: positioning with respect to related works is limited. For example, layer redundancy (which is the opposite of diversity) has been considered in the context of network pruning: https://openaccess.thecvf.com/content_CVPR_2019/papers/He_Filter_Pruning_via_Geometric_Median_for_Deep_Convolutional_Neural_Networks_CVPR_2019_paper.pdf 

review_title: fragile foundations 

Meta-review: The paper looks into generalization performance of NNs in supervised learning setting. The authors propose a regularizer to enhance neuron diversity in each layer(within-layer activation diversity) as a regularizer to improve generalization.  The proposed idea is an extension of Cogswell's work with different regularization terms. The appearance of the term related to the layer output diversity in the generalization bound provides theoretical support for the proposed idea.They use Radamacher complexity as a tool to show this and bound the estimation error.

pros.
-The paper looks into an interesting problem. Designing a regularizer to improve generalization performance of NNs is of huge importance.
-The paper is well-presented and clear.

cons.
-The main drawback of the paper is lack of proper comparison to other regularizers and showing the uniqueness/superiority of this regularizer and how it improves over existing methods either theoretically or with experiments. Without that the significance of this work is limited. 
-The authors response to Reviewer 2's comment was not convincing enough. I encourage the authors to improve this in the next iteration of the paper. 
- i suggest doing a better job in including the related work as also mentioned by the reviewer.
-The experiment section can use more explanation and details on choice of hyper parameters, etc 
- showing performance improvement for a deep architecture would definitely  improve the paper. In the current version only 2 and 3 layer toy examples are shown.
Decision: Reject