Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The author proposed a simple but yet effective technique in order to regularized neural networks. The results obtained are quite good and the technique shows to be effective when it it applied even on state of the art topologies, that is welcome because some regularization techniques used to be applied in easy task or on a initial configuration which results are still far from the best known results. 
review_title: Good results on deep nets

final_score: rating: null
review_content: In the experiments section only very deep networks were chosen to showcase the improved performance when using OrthoReg. Any particular reason for this? How do CNNs with fewer layers perform when using OrthoReg?

The increase is in performance is quite little as compared to the original networks using other regularization schemes. More experimental validation might help. Can you please comment on this.

Biases also play important role in de-correlation of neural responses. What can one expect the learned bias statistics to be in presence of OrthoReg?

Given that the deep networks mostly use rectified non-linearities as activation functions it seems quite natural to avoid penalizing negative correlations. Can you please comment on this.

It was shown previously that orthogonal weight initialization helps in training very deep networks faster. Any such observations were made when using OrthoReg?

review_title: Orthogonality of weight features might not always lead to improved performance

final_score: rating: null
review_content: This is a good paper with the appropriate experimentation. Regularization must be tested on deep and complex topologies, near to state of the art. Other papers test reg. with simple models where regularization helps but are not in the edge...


review_title: Very good results

final_score: rating: 7: Good paper, accept
review_content: Encouraging orthogonality in weight features has been reported useful for deep networks in many previous works. The authors present a explicit regularization cost to achieve de-correlation among weight features in a layer and encourage orthogonality. Further, they also show why and how negative correlations can and should be avoided for better de-correlation. 

Orthogonal weight features achieve better generalization in case of large number of trainable parameters and less training data, which usually results in over-fitting. As also mentioned by the authors biases help in de-correlation of feature responses even in the presence of correlated features (weights). Regularization techniques like OrthoReg can be more helpful in training deeper and leaner networks, where the representational capacity of each layer is low, and also generalize better.

Although the improvement in performances is not significant the direction of research and the observations made are promising.
review_title: Orthogonality of weight features might not always lead to improved performance

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a new regulariser for CNNs that penalises positive correlations between feature weights, but does not affect negative correlations. An alternative version which penalises all correlations regardless of sign is also considered. The paper refers to these as "local" and "global" respectively, which I find a bit confusing as these are very general terms that can mean a plethora of things.

The experimental validation is quite rigorous. Several experiments are conducted on benchmark datasets (MNIST, CIFAR-10, CIFAR-100, SVHN) and improvements are demonstrated in most cases. While these improvements may seem modest, the baselines are already very competitive as the authors pointed out. In some cases it does raise some questions about statistical significance though. More results with the global regulariser (i.e. not just on MNIST) would have been interesting, as the main novelty in the paper seems to be leaving the negative correlations alone, so it would be interesting to see exactly how much of a difference this makes.

One of my main concerns is ambiguity stemming from the fact that the paper sometimes discusses activations and sometimes filter weights, but refers to both as "features". However, the authors have already said they will address this.

The paper somewhat ignores interactions with the choice of nonlinearity, which seems like it could be very important; especially because the goal is to obtain feature activations that are uncorrelated, and this is done only by applying a penalty to the weights (i.e. in a data-agnostic way and also ignoring any nonlinearity). I believe the authors already mentioned in their responses to reviewer questions that this would be addressed, but I think this important and it definitely needs to be discussed.

In response to the authors' answer to my question about the role of biases: as they point out, it is perfectly possible to combine their proposed technique with the "multi-bias" approach, but this was not really my point. Rather, the latter is an example that challenges the idea that features should not be positively correlated / redundant, which seems to be the assumption that this work is built upon. My current intuition is that it's okay to have correlated features, as long as you're not wasting model capacity on them. This is the case for "multi-bias", seeing as the weights are shared across sets of correlated features.

The dichotomy between regularisation methods that reduce capacity and those that don't which is described in the introduction seems a bit arbitrary to me, especially considering that weight decay is counted among the former and the proposed method is counted among the latter. I think this very much depends on ones definition of model capacity (clearly weight decay does not actually reduce the number of parameters in a model).

Overall, the work is perhaps a bit incremental, but it seems to be well-executed. The results are convincing, even if they aren't particularly ground-breaking.
review_title: Solid experimental validation

final_score: rating: null
review_content: Have you try to test with several crops as Graham did in his paper of Fractional Max-Pooling? He arrive to 3.47% with 100 test on CIFAR-10.




review_title: Regularization helps

final_score: rating: null
review_content: The paper seems to conflate orthogonality and correlation of the feature weights with orthogonality and correlation of the feature activations in some places. Could you clarify which one of these is meant in e.g. Section 2.2?

These are not equivalent because there is always an interaction with the nonlinearity. In the ReLU case for example, relu(Wx) and relu(-Wx) will be perfectly orthogonal (one is zero iff the other is nonzero and vice versa), but the weights are negatively correlated.

As AnonReviewer1 points out, there is a similar interaction with the bias parameters, which is ignored when only looking at the correlation of the weights. In fact, some authors have found it useful to have multiple feature detectors with exactly the same weights, differing only in their bias: https://arxiv.org/abs/1604.00676 So it's not immediately obvious to me that having only uncorrelated / negatively correlated weights is a good thing, could you comment on this as well?
review_title: activations vs. weights

Meta-review: The paper presents a new regularization approach for deep learning that penalizes positive correlations between features in a network. The experimental evaluation is solid, and suggests the proposed regularized may help in learning better convolutional networks (but the gains are relatively small).
Decision: Accept (Poster)