Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces "projected error function regularization loss" or PER, an alternative to batch normalization. PER is based on the Wasserstein metric. The experimental results show that PER outperforms batch normalization on CIFAR-10/100 with most activation functions. The authors also test their method on language modeling tasks.

Caveat: I'm not an expert in this domain. Hence, please take my rating with a large grain of salt.

Comments/questions:
- What's the computational cost of using PER over batch norm?  
- Related to my other question: For the CIFAR-10 & CIFAR-100 comparison. What was the training time for BN vs PER?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This submission belongs to the general field of neural networks and sub-field of activation regularisation. In particular, this submission proposes a novel approach for activation regularisation whereby a distribution of activations within minibatch are regularised to have standard normal distribution. The approach, projected error function regularisation (PER), accomplishes that by minimising an upper-bound on 1-Wasserstein distance between empirical and standard normal distributions. 

I think the idea described in this submission is interesting. Unfortunately, I have issues with 1) presentation, 2) experimental results, 3) English. 

The PER is presented as an objective function that minimises an upperbound on 1-Wasserstein. I believe I have seen no evidence to the origin of PER other than it is the upper-bound on 1-Wasserstein. Therefore, I find it strange to see a presentation where first an objective function is introduced, then 1-Wasserstein is described, and after applying standard inequality you obtain an expression that is PER. The current presentation seems to indicate that before this derivation has been done no one new the connection between PER and the upper bound on 1-Wasserstein. I disagree and say that you obtained the upper bound on 1-Wasserstein and called it PER. For unknown reasons you decided to present first PER, then upper bound and finally claim connection. This is a mistake as it is not a connection but merely a consequence. 

Simply looking up CIFAR-10 best numbers on any search engine I can find significantly better numbers. It is therefore unclear why did you decide to use sub-optimal configuration without commenting on that. The same applies to PTB and possibly to WikiText2. 

There are numerous places where English is not adequate. For instance, "new perspective of concerning the target distribution". 

Following the rebuttal stage where the authors have made significant changes to the manuscript I have decided to increase my assessment score. 



review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new method to normalize activations in neural networks, based on an upper bound of the sliced Wasserstein distance between the empirical activation distribution and a standard Gaussian distribution. I think this feels like a "borderline" case to me. The paper clearly has merits, at the same time there're some issues to be addressed.

Pros:
- The idea is clearly presented.
- Better performance than BN is achieved in many experiments.
- Empirical evidence in Section 4.3 looks good, suggesting the proposed method does do the job as expected. The means and variances stabilize as training progresses.

Cons:
- While the method based on sliced Wasserstein distances sounds new, the novelty seems limited since the idea of whitening the activation distribution to unit Gaussian was introduced before as mentioned by the authors. The paper claims the random projection may capture “interaction between hidden units”, but it seems the method proposed in e.g. Huang et. al. 2018 also has projection matrices that might be doing similar things?

- I’m concerned about the actual computation cost of the proposed method. Although the method does not introduce any additional parameter compared to BN or VCL, it seems to require multiple random projections for each layer (s=256 in the experiments)? This could be much slower than the BN. A clarification/comparison of the wall clock running time would be desirable.

- In terms of the image experiments, I do expect to see results with larger datasets/models, though not absolutely necessary.

Typos:
- Page 5, Eq. 9, x_i should be h_i instead?
- Page 9, beta^l_j = 0 and ??^_j = 1
review_title: Official Blind Review #3

Meta-review: This paper presents an interesting and novel idea that is likely to be of interest to the community. The most negative reviewer did not acknowledge the author response. The AC recommends acceptance.
Decision: Accept (Poster)