Regularizing activations in neural networks via distribution matching with the Wasserstein metric
Authors: Taejong Joo
Donggu Kang
Byunghoon Kim
Venue:  (2020)
Keywords: regularization
Wasserstein metric
deep learning

Abstract: Regularization and normalization have become indispensable components in training deep neural networks, resulting in faster training and improved generalization performance. We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution. PER randomly projects activations onto one-dimensional space and computes the regularization loss in the projected space. PER is similar to the Pseudo-Huber loss in the projected space, thus taking advantage of both $L^1$ and $L^2$ regularization losses. Besides, PER can capture the interaction between hidden units by projection vector drawn from a unit sphere. By doing so, PER minimizes the upper bound of the Wasserstein distance of order one between an empirical distribution of activations and the standard normal distribution. To the best of the authors' knowledge, this is the first work to regularize activations via distribution matching in the probability distribution space. We evaluate the proposed method on the image classification task and the word-level language modeling task.


Published as a conference paper at ICLR 2020
REGULARIZING ACTIVATIONS IN NEURAL NETWORKS VIA DISTRIBUTION MATCHING WITH THE WASSER-
STEIN METRIC
Taejong Joo ESTsoff
Donggu Kang ESTsoft
Byunghoon Kim
Hanyang University
Republic of Korea tjoogestsoft.com com
Republic of Korea y com
Republic of Korea byungk im
hanyang hrannggimeaayyagg.. ac kr
ABSTRACT
Regularization and normalization have become indispensable components in training deep neural networks, resulting in faster training and improved general- ization performance. We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution. PER randomly projects activations onto onc-dimensional space and computes the reg- ularization loss in the projected space. PER is similar to the Pscudo-Huber loss in the projected space, thus taking advantage of both I,a and T.2 regularization losses. Besides, PER can capture the interaction between hidden units by projection vec- tor drawn from unit sphere. By doing so, PER minimizes the upper bound of the Wasserstein distance of order one between an empirical distribution of activations and the standard normal distribution. To the best of the authors' knowledge, this is the first work to regularize activations via distribution matching in the probability distribution space. We evaluate the proposed method on the image classification task and the word-level language modeling task.
INTRODUCTION
Training of deep neural networks is very challenging due to the vanishing and exploding gradient problem (Hochreiter 1998 Glorot Bengio 2010)) the presence of many flat regions and saddle points (Shalev-Shwal Shwartz al. 2017). and the shattered gradient problem (Balduzzi et al. [017)). To remedy these issues, various methods for controlling hidden activations have been proposed such as
normalization (loffe & Szegedy, 2015 Huang al. 2018), regularization (Littwin & Wolf 2018), 2016
initialization (Mishkin & Matas 2016 Zhang et al. 2019), and architecture design (He et al.
Among various techniques of controlling activations, trolling their first and second moments. Back in the 1990s, it has been known that the neural net- work training be benefited from normalizing input statisties that samples have and identity covariance matrix (LeCun et al. 1998) Scchaudolph) 1998). This idea motivated batch normalization (BN) that considers hidden activations the imput the next layer and normalizes scale and shift of the activations (loffe & Szegedy 2015).
well-known and successful path is con-
mean
Recent works show the effectiveness of different sample statistics of activations for normalization and regularization. Deecke et al. (2019) and Kalayeh & Shab (2019) normalize activations to several modes with different scales and translations. Variance constancy loss (VCL) implicitly normalizes the fourth moment by minimizing the variance of sample variances, which enables adaptive mode separation or collapse based on their prior probabilities (Littwin & 2018). BN is also extended to whiten activations (Huang et al. 2018 2019), and to normalize general order of central moment in the sense of L norm including L and L (Liao et al. 2016 Hoffer et al. 2018).
In this paper, propose projected error function regularization (PER) that regularizes activations in the Wasserstein probability distribution space. Specifically, PER pushes the distribution acti- vations to be close the standard normal distribution. PER shares similar strategy with previous approaches that dictates the ideal distribution of activations. Previous approaches, however, deal with single or few sample statistics of activations. On the contrary, PER regularizes the activations

---

Published as a conference paper at ICLR 2020
-2.5
0.0
2.5
-2.5 0.0 2.5
-2.5
0.0
2.5 -2.
0.0
2.
X
(a)
(b)
(c)
(d)
Figure 1: Limitation of statistics in terms of representing the probability distribution. In all subplots, has zero mean and unit variance and y ~ (0, 1). In (a) (x, v) ~ (0, f). In (b), x ~ N 1) but correlated with y. In (c), x follows a skewed distribution. In (d), æ follows a bi-modal distribution. Standardization cannot differentiate (a)-(d) and whitening cannot differentiate (a), (c), and (d).
by matching the probability distributions, which considers different statistics simultancously. c.g., all orders of moments and correlation between hidden units. The extensive experiments on multiple challenging tasks show the effectiveness of PER.
RELATED WORKS
Many modern deep learning architectures employ BN an essential building block for better per- formance and stable training even though its theoretical aspects of regularization and optimization are still actively investigated (Santurkar et al. 2018 Kohler al., 2018 Bjorck al. [0188] Yang
et al. 2019). Several studies have applied the idea of BN that normalizes activations via the sample
mean and the sample variance to a wide range of domains such as recurrent neural network (Lei Ba
et al. 2016 and small batch size training (Wu & He 2018).
Huang et al. (2018) 2019) propose normalization techniques whitening the activation of each layer. This additional constraint on the statistical relationship between activations improves the gener- alization performance of residual networks compared to BN. Although the correlation between activations are not explicitly considered, dropout prevents activations from being activated at the same time, called co- ao-adaptation, by randomly dropping the activations (Srivastava et al., 2014), the weights (Wan et al. 2013), and the spatially connected activations (Ghiasi et al. 2018).
Considering BN as the normalization in the L space, several works extend BN to other spaces, i.c., other norms. Streaming normalization (Liao al. 2016) explores the normalization of different order of central moment with L norm for general p. Similarly. Hoffer et al. (2018) explores L and ~ normalization, which enable low precision computation. Littwin & Woll (2018) 8) proposes regularization loss that reduces the variance of sample variances of activation that is closely related to the fourth moment.
The idea of controlling activations via statistical characteristics of activations also has motivated initialization methods. An example includes balaneing variances of each layer (Glorot & Bengio
2010 He et al., al.,, 2015), bounding scale of activation and gradient (Mishkin & Matas] |016]] Balduzzi al. 2017; Gehring al., 2017] Zhang et al. 2019). and norm preserving (Saxe al. (013)). Although the desired initial state may not be maintained during training, experimental results show
that they can stabilize the learning process as well.
Recently, the Wasserstein metric has gained much popularity in a wide range of applications in deep learning with some nice properties such being metric in a probability distribution space without requiring common supports of two distributions. For instance, it is successfully applied to multi- labeled classification (Frogner et al.) (2055), gradient flow of policy update in reinforcement learning (Zhang et al. 2018), training of generative models (Arjovsky al.) 2017: Gulrajani et al. 2017 language
Kolouri et al. 2019), and capturing long term semantic structure in
model (Chen et al. 2019).
While the statistics such as mean and (co) vcrianrience are useful summaries of a probability distribution, they cannot fully represent the underlying structure of the distribution (Fig. []. Therefore, regular-
2

---

Published as a conference paper at ICLR 2020
izing normalizing activation follow the target distribution via statisties can be ineffective in some cases. For instance, normalizing activations via single mean and variance such as BN and decorrelated BN (Huang al. 2018) can be inadequate in learning multimodal distribution (Bilen & Vedaldi 2017: Deecke et al. 2019). This limitation motivates us to investigate a more general way of regularizing the distribution of activations. Instead of controlling activations via statistics, we define the target distribution and then minimize the Wasserstein distance between the activation distribution and the target distribution.
3 PROJECTED ERROR FUNCTION REGULARIZATION
We consider a neural network with I. layers each of which has di hidden units in layer 1. Let D = {(r;; y:)) be training samples which are assumed probability distribution Px..- In this paper, consider the optimization by stochastic gradient descent with mini-batch of samples randomly drawn from at each training iteration. For i-th element of the samples, the neural network recursively computes:
i.i.d. samples drawn from
h á (w'n i B¹)
(1)
where h x; E RR, h Rd is an activation in layer I, and á an activation function, In the case of recurrent neural networks (RNNs), the recursive relationship takes the form of:
h = & WW + W 1 + bb))
where h², is an activation in layer I time and hy, is initial state. Without loss of gener- ality, we focus activations in layer / of feed-forward networks and the mini-batch of samples {(r;; y1)))=-- Throughout this paper, let f be function made by compositions of recurrent relation in equation | up to layer 1, i.e., i.e., h (xx,, and f/ be j-th output of f!.
This paper proposes a new regularization loss, called projected error function regularization (PER), that encourages activations to follow the standard normal distribution. Specifically, PER directly matches the distribution of activations to the target distribution via the Wasserstein metric. Let f P(IRd)) be the Gaussian measure defined (()) /म. SA exp (- || |1 dæ and uh
>, Ong E P(IRd) be the empirical measure of hidden activations where 5n₃ is the Dirae unit mass on h.. Then, the Wasserstein metric of order between μ and Un is defined by:
1/p
inf W uw) (EIII(A)))) J xxe
dl (xx (dx, dy)
where [1 P/.) the set of all joint probability measures on R4 x R44 having the first and the second marginals y and Phi respectively.
Because direct computation of equation iis intractable, we consider the sliced Wasserstein distance (Rabin al. 2011) approximating the Wasserstein distance by projecting the high dimensional distributions onto R (Fig. 2p. It is proved by that the sliced Wasserstein and the Wasserstein are equivalent metries (Santambrogio) 20155 Bonnotte 2013). The sliced Wasserstein of order one between μ and iha can be formulated as:
SW1(H,))) JSL.
----1 Wi )dX(0)
Fuo (I)
Σ oi5z drd)(0)
(4) where gdi-1 is a unit sphere in dR μi and Vng represent the measures projected to the angle 0, À is uniform measure gd-1, and Fro (x) is cumulative distribution function of μa. Herein, equation can be evaluated through sorting (hh } for each angle O.
While we can directly use the sliced Wasserstein in equation 4 as a regularization loss, it has a com- putational dependency
the batch dimension due to the sorting. The computational dependency between samples may not be desirable in distributed and large-batch training that becoming more and more prevalent in recent years. For this reason, remove the dependency by applying the
3

---

Published as a conference paper at ICLR 2020
Current Distribution
Target Distribution
Projection
Figure 2:
Illustration of minimization of the sliced Wasserstein distance between the current distri- bution and the target distribution. Note that it only concerns a distance in the projected dimension.
Algorithm Backward pass under PER
Input The number of Monte Carlo evaluations s, an activation for i-th sample h;, the gradient of the loss Cn,C, regularization coefficient À
g f-
2: for & <- to s do
3: Sample v ~ N(O, (0, I)
4: 0 - v/
1|
5: Project h + (hg 0)
6: 9k + erf /22
7: g + g/0/8
8: end for
9: return Vn, L + Ag
Minkowski inequality to equation
and obtain the regularization loss (
/
b
SWi,,, -
Fro (2) 1 11(,0)s= dxd)(0)
Iga-1
Ojerf
(1/:00) 0)
(11/
2 exp
(- 2
(h1, 0)2 d2(0) S (uni) (5)
22
3dd
i=1
whose gradient with respect to h is:
Vns (ppe((u)))))) Eg Fa-u(ga-1) errf (00 h2/ 0
(6)
where US$ is the uniform distribution on Sd.-1, this paper, expectation over U U(S is approximated by the Monte Carlo method with number of samples. Therefore, PER results in simple modification of the backward pass as Alg. T
Encouraging activations to follow the standard normal distribution be motivated by the natu- ral gradient (Amari 1998). The natural gradient is the steepest descent direction in a Riemannian manifold, and it is also the direction that maximizes the probability of not increasing generaliza- tion error (Roux et al. 2008). The natural gradient obtained by multiplying the inverse Fisher information matrix the gradient. In Raiko al.) (2012) and Desjardins et al. (2015), under the independence assumption between forward and backward passes and activations between different layers, the Fisher information matrix block diagonal matrix each of which block given by:
0C @vec WI) @vec WI)
OC
T
DC Oc T
F E(a.y)
Ex 11-112111
B(a.y)
Da da
where vec( (Wi) is vectorized W h!-1 f1)1(()), and a' f b' for ~.
Since computing the inverse Fisher information matrix is too expensive to perform every iterations, previous studies put efforts into developing reparametrization techniques, activation functions, and
4

---

Published as a conference paper at ICLR 2020
loss
gradient
Hseudo-Huuerr Huber
0.75
Pseudo-tubber
2.0
H.ber
0.50
1.5
0.25
0.00
1.0
-0.50
-1.00
-2
Figure 3: Illustration of PER and its gradient R. Herein, PER shifted by so that per (0) -0)
The Huber loss is defined as (xx)) = |ar] 0.5 in |a|| and h(x) x²/2 | S and the Pseudo-Huber loss defined as g(x) = vI x 1.
regularization losses make F' close to I, thereby making the gradient close to the natural gradient.
For instance, making zero mean and unit variance activations (LeCun et al. 1998 Schraudolph
1998
Glorot & Bengio, 2010 Raiko et al.) 2012 Wiesler et al. 2014) and decorre lated activations
Cogswell et al.
2016 Xiong et al. 2016 Huang et al. 2018 make E h f-1" I, and these
techniques result in taster traming and improved generalization performance. this perspective, expected that PER will enjoy the same advantages by matching Phi to (0.
3.1 COMPARISON CONTROLLING ACTIVATIONS L SPACE
In this subsection, we theoretically compare PER with existing methods that control activations in L" space. (I) is the space of measurable functions whose p-th power of absolute value is Lebesgue integrable, and norm of (PR is given by:
I 1bb= ( [(()))dP(())) <0
(8)
where Px is the unknown probability distribution generating training samples {x; He Since we have no access Px. it approximated by the empirical measure of mini-batch samples.
The L norm is widely used in the literature for regularization and normalization of neural networks. For instance, activation norm regularization (Merity et al. (017a) penalizes L norm of activations. As another example, BN and its p-th order generalization use L norm such that the norm of the centralized activation, or pre-activation, bounded:
M1 Pj
(( 75(0)))))))))))))))))))))))))> +// 5(hg))
1ik, p3)1)) 1/v
Σx
where hl j-th unit of h. Fi = Σn Me the sample mean, 8/ is a learnable shift parameter, and 73 is learnable scale parameters. Herein, we have || { ||,= 1 for any unit j and any empirical measure, thus || 7/4 - f3 Il, 3/ 11 1751 (83).
PER differs from L norm-based approaches in two aspects. First, PER can be considered as L norm with adaptive order in the projected space because it is very similar the Pseudo-Huber loss in one-dimensional space (Fig. 3). Herein, the Pseudo-Huber loss is smooth approximation of the Huber loss (Huber 1964). Therefore, PER smoothly changes its behavior between L' and L norms. making the regularization loss sensitive to small values and insensitive to outliers with large values. However, the previous approaches use predetermined order p. which makes the norm to change insensitively in the near-zero region when p S or to explode in large value region when p >
Second, PER captures the interaction between hidden units by projection vectors, unlike L norm. :o2 Eis (hh) e where { is the natural basis of To this, let f | 1o Σn, |M,
5

---

Published as a conference paper at ICLR 2020
Top-1 eπror rates of ResNets on CIFAR-IO. Lower is better. All numbers are rounded to two dec- imal places. Boldface indicates the minimum error. and are results from Zhang et al. (2019) and He et al. 2016), respectively.
Table 1:
Table 2:
Top-1 enror rates of 11-layer CNNs on tiny ImageNet. Lower is bet- ter. All numbers rounded to two decimal places. Boldface indicates the minimum error. Numbers in parenthe- ses represent results in Littwin & Wolf
2018
Model
Method
Test error
ResNet-56
Vanilla
7.21
BN
6.95
Method
Test error
PER
6.72
Vanilla
37,45 (39.22)
ResNet-110
Vanilla
6,90 (7.24*)
BN
39.22 (40.02)
BN
6.62 (6.61% **)
VCL
(37.30)
PER 6.19
PER
36.74
Rd: That is, the norm computes the regularization loss, or the normalizer, of activations with the natural basis as a projection vector. However, PER uses general projection vectors U(S*-11), capturing the interaction between hidden units when computing the regularization loss. These two differences make PER more delicate criterion for regularizing activations in deep neural networks than L norm, as we will show in the next section.
EXPERIMENTS
This section illustrates the effectiveness of PER through experiments different benchmark tasks with various datasets and architectures. We compare PER with BN normalizing the first and second moments and VCL regularizing the fourth moments. PER is also compared with L' and L acti- vation norm regularizations that behave similarly in some regions of the projected space. We then analyze the computational complexity PER and the impact of PER the distribution of activations. Throughout all experiments, we use 256 number of slices and the same regularization coefficient for the regularization losses computed in each layer.
4.1 IMAGE CLASSIFICATION IN CIFAR-10, CIFAR-1O0, AND TINY IMAGENET
We evaluate PER in image classification task in CIFAR (rizhevsky et al. 2009) and a subset of ImageNet (Russakovsky et al. 2015), called tiny ImageNet. We first evaluate PER with ResNet (He et al. 2016) in CIFAR-I 10 and compare it with BN and vanilla network initialized by fixup initialization (Zhang et al. 2019). We match the experimental details in training under BN with He et al. 2016)) and under PER and vanilla with Zhang al. (2019), and we obtain similar performances presented in the papers. Herein, we search the regularization coefficient over { 3e--4, le-4, 3e-5, le- J. Table I presents results of CIFAR-10 experiments with ResNet-56 and ResNet-110. PER outperforms BN as well as vanilla networks in both architectures. Especially, PER improves the test errors by 0.49 % and 0.71% in Re and ResNet- 10 without BN, respectively.
We also performed experiments on an 11-layer convolutional neural network (11-layer CNN) ex- amined in VCL (Litwwin & Woll 2018). This architecture originally proposed in Clevert et al. 2016). Following Littwin & Wolf (2018), we perform experiments on 11-layer CNNs with ELU, ReLU, and Leaky ReLU activations, and match experimental details in Littwin Wolf 2018) ex- cept that we used 10x less learning rate for bias parameters and additional scalar bias after ReLU and Leaky ReLUU based on /Zhang et al. (2019). By doing so, we obtain similar results presented in Littwin & Wolf 2018). Again, search space of the regularization coefficient is 3e-4, Ic-4, 3e-5, Ie-5 For ReLU and Leaky ReLU in CIFAR- -00, however, we additionally search 3c-6, 1c-6, 3e-7, le-7 because of divergence of training with PER in these setting. As shown in Table 3 PER shows the best performances on four out of six experiments. other cases, PER gives compatible performances to BN or VCL, giving 0.16 % less than the best performances.
Following Littwin & Wolf (2018). PER also evaluated on tiny ImageNet. this experiment, the number of convolutional filters each layer is doubled. Due to the limited time and resources,

---

Published as a conference paper at ICLR 2020
Table 3: Top-1 error rates of 11-layer CNNs on CIFAR-IO and CIFAR-100. Lower better. All numbers are rounded to two decimal places. Boldface indicates the minimum error. Numbers in parentheses represent results in Littwin & Wolf (018).
Activation
Method
10
CIFAR-100
ReLU
Vanilla
8.43 (8.36)
29.45 (32.80)
BN
7.53 (7.78)
29.13 (29.10)
VCL
7.80 (7.80)
30.30 (30.30)
PER
7.21
29.29
LeakyReLU
Vanilla
6.73 (6.70)
26.50 (26.80)
BN
6.38 (7.08)
26.83 (27.20)
VCL
6.45 (6.45)
26.30 (26.30)
PER
6.29
25.50
ELU
Vanilla
6.74 (6.98)
27.53 (28.70)
BN
6.69 (6.63)
26.60 (26.90)
VCL
6.26 (6.15)
25.86 (25.60)
PER
6.42
25.73
conduct experiments only with ELU that gives good performances for PER, BN, and VCL CIFAR. As shown in Table 2 PER is also effective in the larger model in the larger image classification dataset.
4.2 LANGUAGE MODELING IN PTB AND WIKITEXT2
We evaluate PER in word-level language modeling task in PTB (Mikolov et al. 2010 and WikiText2
(Merity et al. 2017b). We apply PER to LSTM with two layers having 650 hidden units with and without variational dropout (VD) proposed in Gal & Ghahramani (2016). We used the same configurations with Merity et al. (017 and failed reproduce the results Merity al. (2077)) Especially, when we rescale gradient when its norm exceeds 10, we observed divergence or bad performance (almost 2x perplexity compared to the published result). Therefore, we rescale gradient with norm over 0,25 instead of 10 based on the default hyperparameter of the PyTorch word-level language mode _ that is also mentioned in Merity et al. (2017a). We also train the networks for 60 epochs instead of 80 epochs since validation perplexity is not improved after 60 epochs in most cases. In this task, PER is compared with recurrent BN (RBN; Cooijmans et al. 2017) because BN is not directly applicable to LSTM. We also compare PER with L and L activation norm regularizations. Herein, the search space of regularization coefficients
embedding (RE) proposed in [Inan al. (2017) and Press & Wolf (2016). and
PER, L regularization, and L regularization (3e-4, lc-4, 3e-5 }. For L' and 2 penalties in PTB, search additional coefficients over 1e-5, 3c-6, 1c-6, 3e-6, lc-6, 3e-7, lc-7 because the searched coefficients seem to constrain the capacity.
We list in Table 4 the perplexities of methods PTB and WikiText2. While all regularization techniques show regularization effects by giving improved test perplexity, PER gives the best test perplexity except LSTM and RE-VD-LSTM in the PTB dataset wherein PER is the second-best method. We also note that naively applying RBN often reduces performance. For instance, RBN increases test perplexity of VD-LSTM by about in PTB and WikiText2.
4.3 ANALYSIS
In this subsection, we analyze the computational complexity of PER and its impact on closeness to the standard normal distribution the I-layee CNN.
Available in http:///wwt.t..oooooooo//caaaaaa
model

---

Published as a conference paper at ICLR 2020
Table 4: Validation and test perplexities on PTB and WikiTex12. Lower is better. All numbers are rounded to one decimal place. Boldface indicates minimum perplexity.
PTB
WikiTex12
Model
Method
Valid
Test
Valid
Test
LSTM
Vanilla
123.2
122.0
138.9
132.7
L penalty
119.6
114.1
137.7
130.0
L penalty
120.5
115.2
136.0
131.1
RBN
118.2
115.1
156.2
148.3
PER
118.5
114.5
134.2
129.6
RE-LSTM
Vanilla
114.1
112.2
129.2
123.2
L' penalty
112.2
108.5
128.6
122.7
L penalty
116.6
108.2
126.5
123.3
RBN
113.6
110.4
138.1
131.6
PER
110.0
108.5
123.2
117.4
VD-LSTM
Vanilla
84.9
81.1
99.6
94.5
I penalty
84.9
81.5
98.2
92.9
L penalty
84.5
81.2
98.8
94.2
RBN
89.7
86.4
104.3
99.4
PER
84.1
80.7
98.1
92.6
RE-VD-LSTM
Vanilla
78.9
75.7
91.4
86.4
L penalty
78.3
75.1
90.5
86.1
1.2 penalty
79.2
75.8
90.3
86.1
RBN
83.7
80.5
95.5
90.5
PER
78.1
74.9
90.6
85.9
4.3.1 COMPUTATIONAL COMPLEXITY
PER has additional parameters. However, BN and VCL require additional parameters for each channel and each location and channel in every layer, respectively; that is, 2.5K and 350K number of parameters are introduced in BN and VCL in the 11-layer CNN, respectively. In terms of time complexity, PER has the complexity of (bdis) for projection operation each layer On the other hand, BN and VCL have (bds) complexities. In our benchmarking, each training iteration takes 0.071 seconds for a vanilla network, 0.083 seconds for BN, 0.087 for VCL, and 0.093 seconds for PER on single NVIDIA TITAN X. Even though PER requires slightly more training time than BN and VCL, this disadvantage can be mitigated by computation of PER is only required in training and PER does have additional parameters.
4.3.2 CLOSENESS TO THE STANDARD NORMAL DISTRIBUTION
To examine the effect of PER the closeness (0, I). we analyze the distribution of activations in 1 Illlaeer CNN in different perspectives. We first analyze the distribution of a single activation hţ for some unit and layer 7 (Fig. 4. We observe that changes in probability distributions between two consecutive epochs are small under BN because BN bound the L norm of activations into learned parameters. On the contrary, activation distributions under vanilla and PER are jiggled between two consecutive epochs. However, PER prevents the variance explosion and pushes the mean to zero. As shown in Fig. while variances of 1/g under both PER and Vanilla are very high at the beginning of training, the variance keeps moving towards one under PER during training. Similarly, PER recovers biased means of 2/1 and 2/22 at the early stage of learning.
To precisely evaluate closeness to the standard normal distribution, we also analyze (V I), pn)) at each epoch (Fig. 5). Herein, the sliced Wasserstein distance is computed by approximating the Gaussian measure using the empirical measure of samples drawn from (o as in Rabin al. 2011). As similar to the previous result, while BN B 0 and 7 at ini- tial state gives small SW (N, I), Ub)) in early stage of training, PER also effectively control
8

---

Published as a conference paper at ICLR 2020
Variila
20
1.5
PCR
tovits
TET.
Layer2
00
300
00
040
0.2
0.0
-04
0:2
100
Tooch
Epech
121
Variance
Layar
00
0.5
-03
1.2
200
c20
300
200
290
533
400
300
100
200
300
300
332
202
303
501
150
200
300
400
500
PER
Varvili
IN
Melar
Variancce
Layer
-0.4
-.25
010
-0.6
2.50
0.-
1.00
200
100
200
200
200
300
500
100
200
300
500
332
202
502
180
200
300
400
Epech
Spech
Spech
Spcch
Figure 4: Evolution of distributions of Pha, Phc, and Vng for fixed randomly drawn i, A on training set. (a)-(c) represent values (0.25, 0.5, 0.75) quantiles under PER, vanilla, and BN. (d) and (c) represent the sample mean and the sample variance of activations. Variance is clipped at 5 for better visualization.
10.0
PER Vanilla
Vanilla
PER
Vanilla BN
5.0
2.5
0.0
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Epoch
Epoch
Epach
(N 1,3)
(b) SW. (N p)))
SW1 T), ),)
Figure 5: Closeness to N/O,, T) in the Wasserstein probability distribution space.
the distribution without such normalization. This confirms that PER prevents the distribution of activation to be drifted away from the target distribution.
CONCLUSION
We proposed the regularization loss that minimizes the upper bound of the 1-Wasserstein distance between the standard normal distribution and the distribution activations. In image classification and language modeling experiments, PER gives marginal but consistent improvements over methods based on sample statistics (BN and VCL) as well as L' and L activation regularization methods. The analysis of changes in activations' distribution during training verifies that PER can stabilize the probability distribution of activations without normalization. Considering that the regularization loss can be easily applied to wide range of tasks without changing architectures or training strategies

---

Published as a conference paper at ICLR 2020
unlike BN, we believe that the results indicate the valuable potential of regularizing networks in the
probability distribution space as a future direction of research.
The idea of regularizing activations with the metric in probability distribution space can be extended to many useful applications. For instance, one can utilize task-specific prior when determining a target distribution, e.g., the Laplace distribution for making sparse activation. The empirical distri- bution of activations computed by pretrained network can also be used as a target distribution to prevent catastrophic forgetting. In this case, the activation distribution can be regularized that does not drift away from the activation distribution learned in the previous task as different from previous approaches constrains the changes in the the function L space of logits (Benjamin al., 2019).
AcKNOWLEDGMENTS
We would like to thank Min-Gwan Seo, Dong-Hyun Lee, Dongmin Shin, and anonymous reviewers for the discussions and suggestions.
REFERENCES
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251-
276, 1998.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Interna- tional Conference on Machine Learning, 2017.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. In International Conference on Learning Representations, 2019.
Hakan Bilen and Andrea Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint 1 2017.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. In Advances in Neural Information Processing Systems, 2018
Nicolas Bonnotte. Unidimensional and Evolution Methods for Optimal Transportation. PhD thesis, Paris 11, 2013.
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Ding- han Shen, Changyou Chen, and Lawrence Carin. Improving sequence-to-sequence learning via optimal transport. In International Conference on Learning Representations, 2019.
Djoé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In International Conference of Learning Represen- futions, 2016.
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfit- ting in deep networks by decorrelating representations. In International Conference on Learning Representations, 2016.
Tim Cooijmans, Nicolas Ballas, César Laurent, Çağlar Gülçehre, and Aaron Courville. Recurrent batch normalization. In International Conference on Learning Representations, 2017.
Lucas Deecke, lain Murray, and Hakan Bilen. Mode normalization. In International Conference on Learning Representations, 2019.
Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, al. Natural neural networks. In Ad- vances in Neural Information Processing Systems, 2015,
Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learn- ing with Wasserstein loss. In Advances in Neural Information Processing Systems, 2015.
10

---

Published as a conference paper at ICLR 2020
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems, 2016.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence sequence learning. In International Conference on Machine Learning, 2017.
Golnaz Ghiasi, Tsung-Yi Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolu-
tional networks. In Advances Neural Information Processing Systems, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Artificial Intelligence and Statistics, 2010.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein gans. In Advances in Neural Information Processing Systems, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Sepp Hochreiter.
The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge- Based Systems, 6(02): 107-116, 1998.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: Efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, 2018.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardiza- tion towards efficient whitening. In IEEE Conference on Computer Vision and Pattern Recogni- fion, 2019.
Peter Huber. Robust estimation of location parameter. The Annals of Mathematical Statistics, pp. 73-101, 1964.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015,
Mahdi M Kalaych and Mubarak Shah. Training faster by separating modes of variation in batch- normalized models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucehi, Ming Zhou, Klaus Neymeyr, and Thomas Hofmann. Towards theoretical understanding of batch normalization. arXiv preprint 10Xiv:1805.10694, 2018.
Soheil Kolouri, Phillip E. Pope, Charles E. Martin, and Gustavo K. Rohde. Sliced Wasserstein auto-encoders. In International Conference on Learning Representations, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Leon Bottou, Genevieve Orr, and Klaus-Robert Müller. Efficient backprop. In Neural Networks: Tricks of the Trade, pp. 9-50. 1998.

---

Published as a conference paper at ICLR 2020
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Qianli Liao, Kenji Kawaguchi, and Tomaso Poggio. Streaming normalization: Towards simpler and more pollogccall-pllusibee normalizations for online and recurrent learning. arXiv preprint arXiv: 1610. 06/.00666600 160, 2016.
Etai Littwin and Lior Wolf. Regularizing by the variance of the activations' sample-variances. In Advances in Neural Information Processing Systems, 2018.
Stephen Merity, Bryan McCann, and Richard Socher. Revisiting activation regularization for lan- guage mns. In International Conference on Machine Learning, 2017a.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations. 2017b.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Čemockk, and Sanjeev Khudanpur. Recur- rent neural network based language model. In Annual Conference of the International Speech Communication Association, 2010.
Dmytro Mishkin and Jiri Matas. All you need is good init. In International Conference on Learning Representations, 2016.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv: 1608. 05859, 2016.
Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Berot. Wasserstein barycenter and its applica- tion to texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, 2011.
Tapani Raiko, Harri Valpola, and Yann LeCun, Deep learning made easier by linear transformations in perceptrons. In Artificial Intelligence and Statistics, 2012.
Nicolas L Roux, Antoin- Manzagol, and Yoshua Bengio. Topmoumoute online natural gra- dient algorithm. In Advances in Neural Information Processing Systems, 2008
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115 (3):211-252, 2015.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55:58-63, 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances Neural Information Processing Systems, 2018.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Nicol Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical
report, 1998.
Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah. Failures of gradient-based deep learning. In International Conference on Machine Learning, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(9):1929-1958 2014.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International Conference Machine Learning, 2013.
12

---

Published as a conference paper at ICLR 2020
Simon Wiesler, Alexander Richard, Ralf Schlüter, and Hermann Ney. Mean-normalized stochastic gradient for large-scale deep learning. IEEE International Conference on Acoustics, Speech and Signal Processing. 2014.
Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision, 2018.
Wei Xiong, Bo Du, Lefei Zhang, Ruimin Hu, and Dacheng Tao. Regularizing deep convolutional neural networks with structured decorrelation constraint. In IEEE International Conference on Data Mining, 2016.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. In International Conference on Learning Representa- fions, 2019.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2019.
Ruiyi Zhang, Changyou Chen, Chunyuan Li, and Lawrence Carin. Policy optimization as Wasser- stein gradient flows. In International Conference on Machine Learning, 2018.
13


Reviews:
[{"review_id": "ryl86MI0tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission belongs to the general field of neural networks and sub-field of activation regularisation. In particular, this submission proposes a novel approach for activation regularisation whereby a distribution of activations within minibatch are regularised to have standard normal distribution. The approach, projected error function regularisation (PER), accomplishes that by minimising an upper-bound on 1-Wasserstein distance between empirical and standard normal distributions. \n\nI think the idea described in this submission is interesting. Unfortunately, I have issues with 1) presentation, 2) experimental results, 3) English. \n\nThe PER is presented as an objective function that minimises an upperbound on 1-Wasserstein. I believe I have seen no evidence to the origin of PER other than it is the upper-bound on 1-Wasserstein. Therefore, I find it strange to see a presentation where first an objective function is introduced, then 1-Wasserstein is described, and after applying standard inequality you obtain an expression that is PER. The current presentation seems to indicate that before this derivation has been done no one new the connection between PER and the upper bound on 1-Wasserstein. I disagree and say that you obtained the upper bound on 1-Wasserstein and called it PER. For unknown reasons you decided to present first PER, then upper bound and finally claim connection. This is a mistake as it is not a connection but merely a consequence. \n\nSimply looking up CIFAR-10 best numbers on any search engine I can find significantly better numbers. It is therefore unclear why did you decide to use sub-optimal configuration without commenting on that. The same applies to PTB and possibly to WikiText2. \n\nThere are numerous places where English is not adequate. For instance, \"new perspective of concerning the target distribution\". \n\nFollowing the rebuttal stage where the authors have made significant changes to the manuscript I have decided to increase my assessment score. \n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkl5pWjptr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method to normalize activations in neural networks, based on an upper bound of the sliced Wasserstein distance between the empirical activation distribution and a standard Gaussian distribution. I think this feels like a \"borderline\" case to me. The paper clearly has merits, at the same time there're some issues to be addressed.\n\nPros:\n- The idea is clearly presented.\n- Better performance than BN is achieved in many experiments.\n- Empirical evidence in Section 4.3 looks good, suggesting the proposed method does do the job as expected. The means and variances stabilize as training progresses.\n\nCons:\n- While the method based on sliced Wasserstein distances sounds new, the novelty seems limited since the idea of whitening the activation distribution to unit Gaussian was introduced before as mentioned by the authors. The paper claims the random projection may capture “interaction between hidden units”, but it seems the method proposed in e.g. Huang et. al. 2018 also has projection matrices that might be doing similar things?\n\n- I’m concerned about the actual computation cost of the proposed method. Although the method does not introduce any additional parameter compared to BN or VCL, it seems to require multiple random projections for each layer (s=256 in the experiments)? This could be much slower than the BN. A clarification/comparison of the wall clock running time would be desirable.\n\n- In terms of the image experiments, I do expect to see results with larger datasets/models, though not absolutely necessary.\n\nTypos:\n- Page 5, Eq. 9, x_i should be h_i instead?\n- Page 9, beta^l_j = 0 and ??^_j = 1", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJl1kZ14cr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces \"projected error function regularization loss\" or PER, an alternative to batch normalization. PER is based on the Wasserstein metric. The experimental results show that PER outperforms batch normalization on CIFAR-10/100 with most activation functions. The authors also test their method on language modeling tasks.\n\nCaveat: I'm not an expert in this domain. Hence, please take my rating with a large grain of salt.\n\nComments/questions:\n- What's the computational cost of using PER over batch norm?  \n- Related to my other question: For the CIFAR-10 & CIFAR-100 comparison. What was the training time for BN vs PER?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an interesting and novel idea that is likely to be of interest to the community. The most negative reviewer did not acknowledge the author response. The AC recommends acceptance.
Decision: Accept (Poster)