Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors in this paper builds on top of the previous orthogonal filter regularization and add a similar regularization on the feature maps of the CNN. Experiments on CIFAR10 and CIFAR100 show that the proposed technique can slightly improve the generalization. 

I have a few concerns:
(1) Overall, the contribution of this paper is very incremental and experimental results are not significant. Given that the technical contribution is weak, I recommend you add more experiments to make the paper more convincing, e.g., adding comparison on ImageNet.
(2) From your regularization in Eq. (1), it only ensures the filter to be orthogonal rather than orthonormal, i.e., filters are not unit vectors. The naming is problematic.
review_title: Very Incremental

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose  (approximate) orthonormalization of the in-layer features combined with (approximate) orthogonalization of the in-batch features.

They claim (based on evidence on CIFAR-10 and CIFAR-100) that this combination is an effective regularizer.

There is a significant literature of constraining weight matrices to be orthonormal (as it is known to be a good pre-conditioner), but people mostly try hard constraints instead of soft constraints.

There are two drawbacks of the proposed method:
- It is relatively costly to batch-orthogonalize the activations.
- Current best results are achieved with wider architectures (Wide Residual Networks by Zagoruyko and Komodakis) and there is current theoretical evidence that over-parametrization helps optimization.  (On the Optimization of Deep Networks:
Implicit Acceleration by Overparameterization by Arora et al.)

Given the fact that the batch-orthogonalization is quite costly, questionably efficient on wide networks, the baselines on cifar-10 and 100 are not up-to-date (Wide Residual Networks came out in 2016 and reach 3.8% error on cifar-10 and about 20% error on cifar-100), I wonder why this work was not based on that work. For this reason, I am not convinced about the scope of this work.

review_title: Interesting but very limited work

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper seeks to enforce orthonormality over model parameters of deep neural networks.  This is done by introducing two loss terms that reflecting orthonormality over parameters and features, respectively. This paper is easy to follow, but the experiments lack some important things. 

The experiments lack results on large data set. 

How to set lambda_1 and lambda_2?

I think both lambda_1 and lambda_2 should be in [0,1]
review_title: This paper is easy to follow, but the experiments lack some important things. 

Meta-review: Based on the reviews, this paper has not been accepted for presentation at the ICLR workshop. However, the conversation and updates can continue to appear here on OpenReview.
Decision: Reject