WeightAlign: Normalizing Activations by
Weight Alignment
Xiangwei Shi? , Yunqiang Li? , Xin Liu? and Jan van Gemert
Computer Vision Lab
Delft University of Technology, The Netherlands

Abstract—Batch normalization (BN) allows training very deep
networks by normalizing activations by mini-batch sample statistics which renders BN unstable for small batch sizes. Current
small-batch solutions such as Instance Norm, Layer Norm, andGroup Norm use channel statistics which can be computed evenfor a single sample. Such methods are less stable than BN as they
critically depend on the statistics of a single input sample. To
address this problem, we propose a normalization of activation
without sample statistics. We present WeightAlign: a methodthat normalizes the weights by the mean and scaled standard
derivation computed within a filter, which normalizes activations
without computing any sample statistics. Our proposed method is
independent of batch size and stable over a wide range of batch
sizes. Because weight statistics are orthogonal to sample statistics,
we can directly combine WeightAlign with any method for
activation normalization. We experimentally demonstrate thesebenefits for classification on CIFAR-10, CIFAR-100, ImageNet,for semantic segmentation on PASCAL VOC 2012 and fordomain adaptation on Office-31.

I. I NTRODUCTION
Batch Normalization [1] is widely used in deep learning.
Examples include image classification [2], [3], [4], object
detection [5], [6], [7], semantic segmentation [8], [9], [10],generative models [11], [12], [13], etc. It is fair to say that for
optimizing deep networks, BatchNorm is truly the norm.
BatchNorm stabilizes network optimization by normalizing the activations during training and exploits mini-batch
sample statistics. The performance of the normalization thus
depends critically on the quality of these sample statistics.
Having accurate sample statistics, however, is not possible
in all applications. An example is domain adaptation, where
the statistics of the training domain samples do match thetarget domain statistics, and alternatives to BatchNorm areused. [14], [15]. High resolution images are another example,
as used in object detection [16], [5], [7], segmentation [8],[9] and video recognition [17], [18], where only a few or just
one sample per mini-batch fits in memory. For these cases,
it is difficult to accurately estimate activation normalization
statistics from the training samples.
To overcome the problem of unreliable sample statistics,
various normalization techniques have been proposed whichmake use of other statistics derived from samples, such as
layer [19], instance [20] or group [21]. These methods are? The authors contributed equally to this work.

applied independently per sample and achieve good performance, but gather statistics just on a single sample and arethus less reliable than BatchNorm.
In this paper, we propose WeightAlign: normalizing activations without using sample statistics. Instead of sample
statistics, we re-parameterize the weights within a filter toarrive at correctly normalized activations. See Fig. 1 for avisual overview. Our method is based on weight statistics
and is thus orthogonal to sample statistics. This allows us to
exploit two orthogonal sources to normalize activations: thetraditional one based on sample statistics in combination with
our proposed new one based on weight statistics. We have thefollowing contributions.
• WeightAlign: A new method to normalize filter weights.
• Activation normalization without computing sample
statistics.
• Performance independent of batch size, and stable performance over a wide range of batch sizes.
• State-of-the-art performance on 5 datasets in image classification, object detection, segmentation and domain
adaptation.
II. R ELATED W ORK
Network Normalization by Sample Statistics. The ideaof whitening the input to improve training speed [22] canbe extended beyond the input layer to intermediate representations inside the network. Local Response Normalization [4], used in AlexNet, normalizes the activations in a smallneighborhood for each pixel. Batch Normalization (BN) [1]
performs normalization using sample statistics computed overmini-batch, which is helpful for training very deep networks.
BN unfortunately suffers from performance degradation when
the statistical estimates become unstable for small batch-size
based tasks.
To alleviate the small batches issue in BN, Batch Renormalization [23] introduces two extra parameters to correct thestatistics during training. However, Batch Renormalization isstill dependent on mini-batch statistics, and degrades performance for smaller mini-batches. EvalNorm [24] corrects thebatch statistics during inference procedure. But it fails to fully
alleviate the small batch issue. SyncBN [25] handles the small
batch problem by computing the mean and variance acrossmultiple GPUs which is not possible without having multiple
GPUs available.

Align.

.

activations

.
3
2
1

Learnable scale
0

Fig. 1. Overview of WeightAlign (WA): Aligning filter weights allows normalizing channel activations. The weight W of an arbitrary convolutional layer iscomposed with Cout filters with size C × k × k. Within a filter (yellow region in the left side), we first flatten it to a vector and then apply our proposed
WA in Eq. (10) to normalize the filter weights with a introduced learnable parameter γ. Normalizing weights of each filter in forward pass can realize thenormalization of activations within each channel. Details in Section III.

Because small mini-batch sample statistics are unreliable,
several methods [19], [20], [21] perform feature normalization based on channel sample statistics. Instance Normalization [20] performs normalization similar to BN but only for asingle sample. Layer Normalization [19] uses activation normalization along the channel dimension for each sample. Filter
Response Normalization [26] proposes a novel combinationof a normalization that operates on each activation channel ofeach batch element independently. Group Normalization [21]divides channels into groups and computes the mean andvariance within each group for normalization. Local Context
Normalization [27] normalizes every feature based on thefilters in its group and a window around it. These methods
can alleviate the small batch problem to some extent, yethave to compute statistics for just a single sample both during
training and inference which introduces instabilities [28], [29].
Instead, in this paper we cast the activation normalization
over feature space into weights manipulation over parameter
space. Network parameters are independent of any sample
statistics and makes our method particularly well suited as
an orthogonal information source which can compensate forunstable sample normalization methods.
Importance of weights. Proper network weight parameter
initialization [30], [4], [31] is essential for avoiding vanishingand exploding gradients [32]. Randomly initializing weightvalues from a normal distribution [4] is not ideal becauseof the stacking of non-linear activation functions. Properly
scaling the initialization for stacked sigmoid or tanh activation
functions leads to Xavier initialization [33], but it is not valid
for ReLU activations. He et al. [3] extend Xavier initialization [33] and derive a sound initialization for ReLU activations. For hypernets, [34] developed principled techniques forweight initialization to solve the exploding activations evenwith the use of BN. In [35] a data-dependent initialization
is proposed to mimic the effect of BN to normalize thevariance per layer to one in the first forward pass. To address
the initialization problem of residual nets, Zhang et al. [36]propose fixup initialization. We draw inspiration from theseweight initialization methods to derive our weight alignment
for activation normalization.
Weight decay [37] adds a regularization term to encourage
small weights. Max-norm regularization is used in [38] toavoid extremely large weights. Instead of introducing an extra
term to the loss, Weight Norm [39] re-parameterizes weights
by dividing its norm to accelerate convergence. Such a reparameterization, however, may greatly magnify the magnitudes of input signals. In contrast, we scale each filter’s
weights as inspired by He et al. [3]’s weight initialization approach, to generate activations with zero mean and maintained
variance. This allows us to normalize activations without using
any sample statistics.
III. P ROPOSED M ETHOD
Batch Normalization (BN) [1] normalizes the features in asingle channel to a zero mean and unit variance distribution, as
in Fig. 2, to stabilize the optimization of deep networks. The
normalized features x̂ are computed channel-wise at training
time using the sample mean µβ and standard derivation σβ
over the input features x as:x̂ =x − µβ,σβ(1)where µβ and σβ are functions of sample statistics of x. For
each activation x̂, a pair of trainable parameters γ, β are
introduced to scale and shift the normalized value thatr = γx̂ + β.

(2)

BN uses the running averages of sample statistics within eachmini-batch to reflect the statistics over the full training set.
A small batch leads to inaccurate estimation of the sample
statistics, thus using small batches degrades the accuracy
severely.
In the following, we express the sample statistics in termsof filter weights to eliminate the effect of small batch, andre-parameterize the weights within each filter to realize thenormalization of activations within each channel.
A. Expressing activation statistics via weights
For a convolution layer, let x be a single channel output
activation, Y be the input activations, and w, b be thecorresponding filter and bias scalar. Specifically, a filter w
consists of c channels with a spatial size of k × k. To
normalize activations by using weights, we show how thesample statistics µβ and σβ in Eq. (1) are expressed in termsof filter weights.

An individual response value x(t) in x is a sum of the
product between the values in w and the values in a co-locatedk 2 c-by-1 vector y(t) in Y, which is equivalently written as
the dot product of filter w and vector y(t),x(t) = w · y(t) + b.

(3)

By computing each individual response x(t) at any location t,
we obtain the full single channel output activation x.
We use random variables x, Y and w to present each of theelements in x, Y and w respectively. Following [34], [33],[3], we make the following assumptions about the network:(1) The w , Y and b are all independent of each other. (2) The
bias term b equals 0.
Expressing µβ of Eq. (1) in terms of weights: The mean ofactivation x can be equivalently represented via filter weightsas:µβ = E[x] = nE[wY ] = nE[w]E[Y ],(4)
where E is the expected value and n = k 2 c denotes the number
of weight values in a filter.
Expressing σβ of Eq. (1) in terms of weights: The varianceof activation x is:
σβ2 = Var[x] =nVar[wY ] = n(E[w2 Y 2 ] − E2 [wY ])=n(E[w2 ]E[Y 2 ] − E2 [w]E2 [Y ])(5)With the Eq. (4) and Eq. (5), we will manipulate the weights
to normalize activations in the following sections.

This shows the relationship of variance between the previous
layer’s output before the non-linearity Z and the current layer’s
single channel activation x.
Our goal is to manipulate the weights to keep the activation
variance of all layers normalized. Thus we relate the varianceof this layer σβ2 with the variance in the previous layer Var[Z],which can be achieved in terms of the weights by setting
1
2 nVar[w] in Eq. (8) to a proper scalar (e.g. 1):
1
nVar[w] = 1.
(9)2
With Eq. (6) and Eq. (9), we can re-parameterize a single
filter weights to have zero mean and a standard deviationp2/n, which achieves the activation normalization like BN
in Eq. (1). Thus, here we propose WeightAlign as,
w − E[w],ŵ = γ pn/2 · Var[w](10)

where γ is a learnable scalar parameter similar to the scaleparameter in BN [1] like Eq. (2). Since WA manipulates theweights, it does not have the β term as in the form of BN,
which is applied on activations directly. The detail explanation
about this is given in supplementary material. From Eq. (10),we can tell that the proposed WeightAlign(WA) method does
not rely on sample statistics computed over a mini-batch,which makes it independent of batch size.
C. Initialization of weights

B. WeightAlign
We normalize activations by encouraging the mean µβ of theactivations in Eq. (4) to be zero, and maintaining the varianceof the activations to be the same in all layers. Inspired by [3],we manipulate the weights to satisfy these requirements.
The mean of the activations µβ in Eq. (4) is forced to be
zero when the weight w in a filter has zero mean:
E[w] = 0.

(6)

We can exploit E[w] = 0 in Eq. (5) to further simplify thevariance of the activations σβ2 in terms of the variance of theweights as,σβ2 = Var[x] = nE[w2 ]E[Y 2 ] = nVar[w]E[Y 2 ].

(7)

Thus, the activation variance σβ2 depends on the variance of
the weights w and on the current layer input Y . To simplify
further, we follow [3] and assume a ReLU activations function.
We use Z to define the outputs of the previous layer before
the activation function, where Y = ReLU(Z).
If the weight of the previous layer has a symmetric distribution around zero then Z has a symmetric distribution aroundzero.1 Because the ReLU sets all negative values to 0, thevariance is halved [3], and we have that E[Y 2 ] = 21 Var[Z].
Substituted into Eq. (7), we have
σβ2 = Var[x] =1

1nVar[w]Var[Z].
2See proof in the supplementary material.

(8)

We initialize each filter weights w to be a zero-meansymmetric
whose standard deviation
p Gaussian distribution
2
2/n
where
n
=
k
c.
For
the first layer, we should(std) isphave 1/n since the ReLU activation is not applied on the
input. But the factor 1/2 can be neglected if it just exists
p on
one layer and for simplicity, we use the same deviation 2/nfor initialization.
D. Empirical analysis and examples
To show the distributions of channel activations using different methods, we build an 8-layer deep network containing 7
convolutional layers and one classification layer with ReLU
as nonlinear activation function. A mini-batch of size 128
independent data is sampled from N (0, 1) is used as input.
Specifically, the standard Kaiming initialization [3] withoutbias term is used to initialize the weights. Fig. 2 shows theactivation distributions of 8 different channels taken fromtwo layers, the 3rd intermediate convolutional layer and thelast classifier layer before the softmax loss. Each channel ofthe last classifier layer represents one class. To exclude thetraining effect, we first use the initialization model to conductcomparison between sample statistics based normalization and
our WA normalization. For the visualization of trained model,
please see Section IV.
For the baseline model, it can be seen in Fig. 2(a) thatthe activation distributions in the intermediate layer startdrifting, leading to a constant output for the classifier layer(i.e. the ’Blue’ indexed class/channel). From Fig. 2(b) of Batch

3rd layerclassifier layer(a) Baseline(b) BN(c) GN(d) WA(e) WA+GN

Fig. 2. Our WA method can alleviate internal covariate shift of activation (see Section III-D for details). Each color represents the activation distribution ofdifferent channels for different layers. For baseline model, the ’Blue’ indexed channel will dominate all other channels, leading to a constant classification
result. Normalizing channel activation in intermediate layer can avoid the constant output for the final classifier layer, which can be realized by WA.

Norm (BN) [1], we note that reducing the Internal Covariate
Shift (ICS) in intermediate layer can alleviate the distribution
drifting for the final classifier layer. The output of the classifierlayer is no longer a constant function. Group Norm(GN) [21]can also reduce ICS to some extent, as shown in Fig. 2(c).
As shown in Fig. 2(d), WA can realize similar functionalityof BN. Our proposed WeighAlign (WA) re-parameterizes theweights within each filter in Eq. (10) to normalize channel
activation. Since weight statistics is orthogonal to sample
statistics, WA can be used in conjunction with BN, GN, LN
and IN, as shown in Fig. 2(e). Please refer to supplementarymaterial for visualization of other normalization methods.
IV. E XPERIMENTS
We show extensive experiments on three tasks and fivedatasets: image classification on CIFAR-10, CIFAR-100 [40]and ImageNet; domain adaptation on Office-31; and semantic
segmentation on PASCAL VOC 2012 [41] where we evaluate
VGG [31] and residual networks as ResNet [30].
A. Datasets
CIFAR-10 & CIFAR-100. CIFAR-10 and CIFAR-100 [40]consist of 60,000 32×32 color images with 10 and 100 classes,
respectively. For both datasets, 10,000 images are selected
as the test set and the remaining 50,000 images are used
for training. We perform image classification experiments andablation study on these datasets.
ImageNet. ImageNet [42] is a classification dataset with 1,000
classes. The size of the training dataset is around 1.28 million,
and 50,000 validation images are used for evaluation.
Office-31. Office-31 [43] is a dataset for domain adaptation
with 4,652 images with 31 categories. The images are collected from three distinct domains: Amazon (A), DSLR (D),and Webcam (W). The largest domain, Amazon, has 2,817
labeled images. The 31 classes consists of objects commonly

encountered in office settings, such as file cabinets, keyboards
and laptops.
PASCAL VOC 2012. PASCAL VOC 2012 [41] contains asemantic segmentation set and includes 20 foreground classes
and a single background class. The original segmentation
dataset contains 1,464 images for training, and 1,499 images
for evaluation. We use the augmented training dataset including 10,582 images [44].
B. Implementation details
We use Stochastic Gradient Descent(SGD) in all experiments with a momentum of 0.9 and a weight decay of 5×10−4 .
The plain VGG model is initialized with Kaiming initialization [3] and ResNets are initialized with Fixup [36]. We do
not apply WA in the final classifier layer. For classification,
we train the models with data augmentation, random horizontal
flipping and random crop, as in [45]. Further implementation
details are given in each subsection.
C. Experiments on CIFAR-10 & CIFAR-100
Comparing normalization methods. We compare our proposed WeightAlign (WA) against various activation normalization methods, including Batch Normalization (BN) [1], Group
Normalization (GN) [21], Layer Normalization (LN) [19],Instance Normalization (IN) [20]. To demonstrate that WA is
orthogonal to all these approaches, we also show the combination of WA with these activation normalization methods.
We conduct experiments on CIFAR-100 as shown in Table I,
where all models are trained with a batch size of 64. WeightAlign outperforms every normalization method except BNover large batch size. By combining WeightAlign with sample
statistics normalization methods, we can see that weightAlignadds additional information and improves accuracy. Especially,
the GN+WA model achieves comparable performance of BN.
To further show it’s flexibility, we fine-tune a pre-trained
BN model on CIFAR-100 with WA. It achieves 22.38% error

TABLE I
E RROR RATE OF R ES N ET 50 ON CIFAR-100 FOR CLASSIFICATION . WA
OUTPERFORMS EVERY NORMALIZATION METHOD EXCEPT BNOVERLARGE BATCH SIZE . W HEN COMBINED WITH STATISTICSNORMALIZATION METHODS , WA IMPROVES ACCURACY.
CIFAR-100
MethodError

Methoderror

Baseline [36]BN [1]
IN [20]
LN [19]GN [21]

28.43
23.02
25.58
26.78
25.46

WA
BN + WAIN + WA
LN + WA
GN + WA

24.92
22.39
24.63
24.01
23.64

(a) VGG16

Fig. 3. Classification error vs. batch sizes. The models are VGG16 andResNet18 trained on CIFAR-100. The error rates of BN increase rapidly whensmall batch sizes are used. Our proposed method does not rely on the sample
statistics and is independent of batch size dimension. The error rates of WA
are stable over a wide range of batch sizes.

TABLE IIE RROR RATE OF R ES N ET 50 ON CIFAR-10 FOR CLASSIFICATION . W EIGHTSTATISTICS NORMALIZATION (WA) HELPS IN ADDITION TO SAMPLESTATISTICS NORMALIZATION .
CIFAR-10
Batch size 64
Method Error Method Error
Baseline [36]BN [23]IN [20]
LN [19]GN [21]

6.46
4.30
6.49
5.02
4.96

WA
BN+WA
IN+WA
LN+WA
GN+WA

6.21
4.29
6.42
5.12
4.60

Batch size 1
Method Error Method Error
Baseline
BN
IN
LN
GN

7.27
6.91
6.82
5.79

WA
BN+WA
IN+WA
LN+WA
GN+WA

(b) ResNet18

6.61
6.50
5.76
5.51

rate, which is the same as BN+WA model trained from scratch.
This shows the possibility of finetuning WA on a pre-trainedmodels with a sample-based normalization method.
On CIFAR-10 we do the same evaluation for ResNet50.
BN and BN+WA are trained only with batch size of 64,
and other methods are trained with batch size of 1 and 64.
For batch size of 64 and 1, the initial learning rates are
0.01 and 0.001, respectively. The baseline method [36] istrained without any normalization methods. Results are given
in Table II. Because CIFAR-10 is less complex than CIFAR100, the performance gaps between different methods are thus
less obvious. Nevertheless, our method is comparable to othernormalization methods and it allows a batch size of 1, which
is not possible for BN. By combining WA with the samplebased normalization methods, we again observe that weightstatistics normalization helps in addition to sample statistics
normalization.
We also compare WA with Weight Normalization (WN) [39]on CIFAR-100 with VGG16 and ResNet50 networks. For
VGG16, the error rates of WN, WA, GN are 6.8%, 4.8%and
7.4% when the batch size is 1, and the ones of WN, WA,
BN, GN are 8.6%, 5.4%, 5.8% and 11.0% when the batch
size is 64. For ResNet50, WN cannot be trained. We cansee WA and WN are workable regardless of the batch sizefor plain networks and WA outperforms WN in this model.
For residual networks, WA can work alone without any
activation normalization layer, while WN cannot. WN is
only normalized by its norm to accelerate training. Withoutnormalization layer in residual networks, WN cannot restrict

the variance of the activation and the residual structure
merges the activation from two branches, which leads to
exploding gradients problem. Instead, WA normalizes theweights by zero-mean and a scaled standard deviation. This
scale is determined by the size of convolutional filter andrestricts the variance of weights and further maintains thevariance of activations.
Small batch sizes. In Fig. 3, we compare BN and WA withdifferent batch sizes on VGG16 and ResNet18. We train withbatch sizes of 64, 8, 4, 2 images per GPU. In all cases, themeans and variances in the batch normalization layers arecomputed within each GPU and not synchronized. The x-axis
shows different batch sizes and y-axis presents the validation
error rates. From Fig. 3 (a), we observe that the performanceof WA is comparable to that of BN on the plain network withlarge batches. The error rates of BN increase rapidly whenreducing the batch size, especially with batch size smaller
than 4. In contrast, our method focuses on the normalizationof weight distributions which is independent of batch size.
The error rates of our method are stable over different batch
sizes. We can observe similar results in Fig. 3 (b) that our
method performs stable over different batch size on residual
network. It is interesting to see that the performance gap
between WA and BN with ResNet18 is smaller than that with
VGG16, which is due to the residual structure.
Depth of residual networks. We evaluate depth of 18, 34,
50, 101, 152 for residual networks with batch sizes of 1,
64 per GPU. Table III shows the validation error rates on
CIFAR-10. We observe that WA is able to train very deep
networks for both small and large batch sizes.
Visualization of weights and activations in training. To
verify the motivation, in this subsection, we visualize the
distribution of weights and activation by training a ResNet50
model on CIFAR-100. A minibatch of 128 images from
CIFAR-100 is input to the trained network to obtain features
and weights.
How are the filter weights distributed? In Fig. 4, we
visualize how the channel weight distribution changes withthe epochs. Fig. 4 (a) illustrates the baseline ResNet50 model
without using any normalization. When using BN, as shown in(a) Baseline(b) BN(c) WA

Fig. 4. Channel weights distribution of trained ResNet50 versus epoch: (a) channel weights of baseline model. (b) channel weights of BN model. (c) channel
weights of WA model. The x-axis represents the value of weights. The y-axis represents the epoch. From top to bottom, the epoch varies from 1 to 300.
WA makes the distributions of channel weights symmetric around 0. With a symmetric distribution around zero for the weight, the activation can also have asymmetric distribution around zero.

TABLE IIIA PPLYING WA TO VARYING DEPTH MEASURED BY ERROR RATE . WA IS
ABLE TO TRAIN VERY DEEP NETWORKS FOR BOTH SMALL AND LARGEBATCH SIZES .

CIFAR-10
Model (+WA)

Batchsize 1
Error

Batchsize 64
Error

ResNet18
ResNet34
ResNet50
ResNet101
ResNet152

5.65
5.84
6.61
5.74
6.09

6.23
5.78
6.42
6.41
6.52

Fig. 4 (b), the weight distribution tends to be symmetric aroundzero. From Fig. 4 (c), we note that the proposed WA method
can realize the same functionality of BN making the weights
symmetric and smooth during training. Channel and layer
weights distribution during training of other normalization
methods can be found in the supplementary material. Witha symmetric distribution around zero for the weight, theactivation can also have a symmetric distribution around zero.
How are the channel activations distributed? In Fig. 2 wehave found that before training, the proposed WA method
can effectively normalize channel activations by aligning theweights within each filter. In Fig. 5 we further explore how
the channel activations distributed on a trained ResNet50
network. We visualize the activation distributions for 4
different channels taken from two layers, a middle layer (the2rd conv layer of the 8th residual block) and the last classifierlayer. Since the weight decay is used, the scale and shiftparameters γ, β are converged to small values. From Fig. 5(d), we have similar observations as in Fig. 2 that our method
can normalize activation of different channels to zero meanand same variance. Fig. 5 (e) also illustrates WA can be usedin conjunction with GN to further normalize the activations.
Functionality of different components. We align thefilter weights by subtracting the mean and dividing by aproperly scaled derivation. For our method, there are twocomponents: E[w] = 0 in Eq. (6) Var[w] = n2 in Eq. (9).
We conduct an comparative results for our model withTABLE IVC OMPARATIVE RESULTS FOR VGG16 AND R ES N ET 18 MODELS WITH
DIFFERENT COMPONENTS . B OTH COMPONENTS CONTRIBUTE
SUBSTANTIALLY TO THE PERFORMANCE .
Components in WA
2
E[w] = 0Var[w] = n
×
X
×
X

×
×
X
X

VGG16
Error

ResNet18
Error

35.74
27.85

28.23
27.97
28.03
24.92

different components for VGG and ResNet18 models on
CIFAR-100. In Table IV we start with the baseline model [36]and augment incrementally with the two components. We
observe that both components contribute substantially to theperformance of the whole model.
D. Comparison with state-of-the-art
Image classification on ImageNet. We experiment witha regular batch size of 64 images per GPU on plain andresidual networks. The total training epoch is 100. The initial
learning rate is 0.1 and divided by 10 at the 30th, 60th and90th epoch. Table V shows the top-1 and top-5 error rates
of image classification on ImageNet. For plain networks,
TABLE VT OP -1 AND TOP -5 ERROR RATES OF IMAGE CLASSIFICATION ON
I MAGE N ET. WA AND WA+BN CAN WORK WELL ON LARGE ANDCOMPLEX DATASETS .
model

Top-1 (%) Error

Top-5 (%) Error

VGG16∗ (Baseline)
VGG16 (BN)∗VGG16 (WA)VGG16 (BN+WA)
ResNet50 (Baseline)† [36]ResNet50 (BN)∗ResNet50 (WA)ResNet50 (BN+WA)

31.30
29.58
29.78
27.07
27.60
24.89
26.62
24.04

11.19
10.16
10.23
8.78
7.71
8.91
7.12

The ∗ denotes that we directly test the PyTorch pretrained model.
The † denotes the numbers from the reference.

WA performs relatively well as BN, only 0.2% difference onintermediate layerclassifier layer(a) Baseline(b) BN(c) GN(d) WA(e) WA+GN

Fig. 5. Channel activation distributions of trained models on CIFAR-100. Different color represents different channel. γ and β are the parameters in eachnormalization. WA can align the channel activations by normalizing channel weights.

top-1 error rate. Combining WA with BN outperforms BN
on VGG16 by 2.51% top-1 error rate and 1.38% top-5 error
rate. For residual networks, WA performs slightly worse thanBN, but combined BN+WA surpasses BN by 0.85% on top-1
error rate and 0.59% on top-5 error rate. This shows that WA
and WA+BN can work well on large and complex datasets.
Domain adaptation on Office-31. Table VI presents
overall scores of different normalization methods and their
combination with WA on Office-31 dataset. Here we use
ResNet50 models pretraied on CIFAR-100, which are obtained
from previous experiments. To perform domain adaptation
experiments, we finetune the pretrained model on a source
domain, and validate it on two target domains with 0.001
initial learning rate, e.g. finetune on A, and validate on W and
D. Model with WA consistently outperforms model with GN,
IN and LN. Table VI also shows that our method WA canbe applied to other normalization methods successfully andimproves domain adaptation performance. In Office-31, thethree domains A, W and D have different sample distribution.
Thus the normalization methods that rely on sample statistics
perform worse than our method WA, which only applies
on weights and is independent of sample statistics. When
combined with WA, other activation normalization methods
can benefit from the sample-independent merit of WA, which
boosts their domain adaptation performance.
Semantic segmentation on PASCAL VOC 2012. To
investigate the effect on small batch size, we conduct
experiments on semantic segmentation with PASCAL VOC
2012 dataset. We select the DeepLabv3 [46] as baselinemodel in which the ResNet50 pre-trained on ImageNet isused as backbone. Given the pretrained ResNet50 models
with BN and BN+WA on ImageNet, we finetune them on
PASCAL VOC 2012 for 50 epochs with batch size 4, 0.007
learning rate with polynomial decay and multi-grid (1,1,1),

TABLE VIC LASSIFICATION ACCURACIES (%) ON O FFICE -31 DATASET (R ES N ET 50).
WA CAN BE APPLIED TO OTHER NORMALIZATION METHODS AND
IMPROVES DOMAIN ADAPTATION PERFORMANCE .
MethodA → W W → A A → D D → A W → D D → W Avg

WA
34.62
BN [1]
46.92
BN + WA 49.23
GN [21] 46.92
GN + WA 46.15
IN [20]36.92
IN + WA 39.23
LN [19]37.69
LN + WA 50.77

42.89
48.91
36.38
42.89
54.46
33.98
41.45
29.89
42.89

52.05
57.53
58.90
50.68
50.68
41.10
38.36
45.20
56.16

44.09
37.11
40.00
39.76
37.83
37.11
38.07
33.97
41.69

87.67
90.41
91.78
82.19
87.67
84.93
90.41
79.45
91.78

80.00
70.00
76.92
75.38
78.46
76.92
73.85
69.23
80.77

56.89
58.48
58.87
56.30
59.21
51.83
53.56
49.24
60.68

and evaluate the performance on the validation set with
output stride 16. The evaluation mIoU of BN method is73.80%, and that of BN+WA is 74.87%. Combined with WA,
the original BN performance is improved by 1.03%. Thisfurther shows that WA improves other normalization methods.

V. C ONCLUSION
We propose WeightAlign; a method that re-parameterizes
the weights by the mean and scaled standard derivation computed within a filter. We experimentally demonstrate WeightAlign on five different datasets. WeightAlign does not relyon the sample statistics, and performs on par with Batch
Normalization regardless of batch size. WeightAlign can also
be combined with other activation normalization methods andconsistently improves on Batch Normalization, Group Normalization, Layer Normalization and Instance Normalization onvarious tasks, such as image classification, domain adaptationand semantic segmentation.

R EFERENCES[1] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” arXiv preprint
arXiv:1502.03167, 2015. 1, 2, 3, 4, 5, 7
[2] T.-H. Chan, K. Jia, S. Gao, J. Lu, Z. Zeng, and Y. Ma, “Pcanet: A simple
deep learning baseline for image classification?” IEEE transactions on
image processing, vol. 24, no. 12, pp. 5017–5032, 2015. 1[3] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:
Surpassing human-level performance on imagenet classification,” inProceedings of the IEEE international conference on computer vision,
2015, pp. 1026–1034. 1, 2, 3, 4
[4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105. 1, 2
[5] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” inProceedings of the IEEE international conference on computer vision,
2017, pp. 2961–2969. 1[6] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only lookonce: Unified, real-time object detection,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2016, pp. 779–
788. 1[7] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time
object detection with region proposal networks,” in Advances in neural
information processing systems, 2015, pp. 91–99. 1[8] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in Proceedings of the IEEE conference on
computer vision and pattern recognition, 2015, pp. 3431–3440. 1[9] H. Noh, S. Hong, and B. Han, “Learning deconvolution network
for semantic segmentation,” in Proceedings of the IEEE international
conference on computer vision, 2015, pp. 1520–1528. 1[10] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
for biomedical image segmentation,” in International Conference on
Medical image computing and computer-assisted intervention. Springer,
2015, pp. 234–241. 1[11] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” arXiv
preprint arXiv:1701.07875, 2017. 1[12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” inAdvances in neural information processing systems, 2014, pp. 2672–
2680. 1[13] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation
learning with deep convolutional generative adversarial networks,” arXiv
preprint arXiv:1511.06434, 2015. 1[14] W.-G. Chang, T. You, S. Seo, S. Kwak, and B. Han, “Domainspecific batch normalization for unsupervised domain adaptation,” inProceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2019, pp. 7354–7362. 1[15] Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou, “Revisiting batch normalization for practical domain adaptation,” arXiv preprint arXiv:1603.04779,
2016. 1[16] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international
conference on computer vision, 2015, pp. 1440–1448. 1[17] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a newmodel and the kinetics dataset,” in proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308. 1[18] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning
spatiotemporal features with 3d convolutional networks,” in Proceedingsof the IEEE international conference on computer vision, 2015, pp.
4489–4497. 1[19] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv
preprint arXiv:1607.06450, 2016. 1, 2, 4, 5, 7
[20] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The
missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022,
2016. 1, 2, 4, 5, 7
[21] Y. Wu and K. He, “Group normalization,” in Proceedings of theEuropean Conference on Computer Vision (ECCV), 2018, pp. 3–19. 1,
2, 4, 5, 7
[22] Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient
backprop,” in Neural networks: Tricks of the trade. Springer, 2012,
pp. 9–48. 1[23] S. Ioffe, “Batch renormalization: Towards reducing minibatch dependence in batch-normalized models,” in Advances in neural information
processing systems, 2017, pp. 1945–1953. 1, 5

[24] S. Singh and A. Shrivastava, “Evalnorm: Estimating batch normalization
statistics for evaluation,” in Proceedings of the IEEE International
Conference on Computer Vision, 2019, pp. 3633–3641. 1[25] C. Peng, T. Xiao, Z. Li, Y. Jiang, X. Zhang, K. Jia, G. Yu, and J. Sun,
“Megdet: A large mini-batch object detector,” in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, 2018,
pp. 6181–6189. 1[26] S. Singh and S. Krishnan, “Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks,” inProceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, 2020, pp. 11 237–11 246. 2[27] A. Ortiz, C. Robinson, D. Morris, O. Fuentes, C. Kiekintveld, M. M.
Hassan, and N. Jojic, “Local context normalization: Revisiting local normalization,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, 2020, pp. 11 276–11 285. 2[28] W. Shao, T. Meng, J. Li, R. Zhang, Y. Li, X. Wang, and P. Luo,“Ssn: Learning sparse switchable normalization via sparsestmax,” inProceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2019, pp. 443–451. 2[29] J. Yan, R. Wan, X. Zhang, W. Zhang, Y. Wei, and J. Sun, “Towards stabilizing batch statistics in backward propagation of batch normalization,”
ICLR, 2020. 2[30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778. 2, 4[31] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
2, 4[32] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training
recurrent neural networks,” in International conference on machinelearning, 2013, pp. 1310–1318. 2[33] X. Glorot and Y. Bengio, “Understanding the difficulty of training
deep feedforward neural networks,” in Proceedings of the thirteenthinternational conference on artificial intelligence and statistics, 2010,
pp. 249–256. 2, 3[34] O. Chang, L. Flokas, and H. Lipson, “Principled weight initialization
for hypernetworks,” in International Conference on Learning Representations, 2020. 2, 3[35] D. Mishkin and J. Matas, “All you need is a good init,” arXiv preprint
arXiv:1511.06422, 2015. 2[36] H. Zhang, Y. N. Dauphin, and T. Ma, “Fixup initialization: Residual
learning without normalization,” arXiv preprint arXiv:1901.09321, 2019.
2, 4, 5, 6[37] A. Krogh and J. A. Hertz, “A simple weight decay can improvegeneralization,” in Advances in neural information processing systems,
1992, pp. 950–957. 2[38] N. Srebro and A. Shraibman, “Rank, trace-norm and max-norm,” inInternational Conference on Computational Learning Theory. Springer,
2005, pp. 545–560. 2[39] T. Salimans and D. P. Kingma, “Weight normalization: A simple
reparameterization to accelerate training of deep neural networks,” inAdvances in Neural Information Processing Systems, 2016, pp. 901–
909. 2, 5[40] A. Krizhevsky, G. Hinton et al., “Learning multiple layers of featuresfrom tiny images,” Citeseer, Tech. Rep., 2009. 4[41] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge 2007 (voc2007)
results,” 2007. 4[42] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet:
A large-scale hierarchical image database,” in 2009 IEEE conference on
computer vision and pattern recognition. Ieee, 2009, pp. 248–255. 4[43] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting visual categorymodels to new domains,” in Proceedings of the 11th European Conference on Computer Vision: Part IV, ser. ECCV’10. Berlin, Heidelberg:
Springer-Verlag, 2010, p. 213–226. 4[44] B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik, “Semantic
contours from inverse detectors,” in 2011 International Conference on
Computer Vision. IEEE, 2011, pp. 991–998. 4[45] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasingdata augmentation,” arXiv preprint arXiv:1708.04896, 2017. 4[46] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoderdecoder with atrous separable convolution for semantic image segmentation,” in Proceedings of the European conference on computer vision(ECCV), 2018, pp. 801–818. 7

A PPENDIX
VI. P ROOF OF SYMMETRIC
Given two independent random variable X and Y , thedistribution of product random variable Z, where Z = XY ,can be found as follows,Z ∞1zfZ (z) =fX (t)fY ( ) dt.
(11)
|t|
t
−∞

p
lead to similar performance. But our scale n/2 has the best
performance. Any other scale factors would cause the failureof training. Thus a proper scaled derivation plays an importantrole to make the training stable.

If the distribution of X is continuous at 0, then we have,
P (Z ≤ z) = P (XY ≤ z)zz
= P (Y ≤ |X > 0)P (X > 0) + P (Y ≥ |X < 0)P (X < 0)
X
X
Z ∞
Z 0
z
z
P (Y ≤ )fX (t) dt +=P (Y ≥ )fX (t) dt.
t
t
0
−∞
(12)
We take the derivation of both sides w.r.t. z and we get,
Z 0
Z ∞
z
−1
z
1
fY ( )fX (t) dt +fY ( )fX (t) dt
fZ (z) =
t
t
t
t
−∞
Z0 ∞
1
z
=
fX (t)fY ( ) dt.
Fig. 6. Ablation study of scale factor. Validation error rates with different|t|t−∞scale factors on CIFAR-100 (ResNet18). A slightly different scale factor can(13) cause a failure of training like the red line. A proper scale factor is important
Suppose the distribution of random variable Y is symmetric to stabilize the optimization. Our scale achieves the best performance among
other trainable scale factors.
around zero, where fY (y) = fY (−y). Then we can have,Z ∞B. Empirical analysis in Section III-Dz
1
fX (t)fY ( ) dt
fZ (z) =We further show activation distributions for different nor|t|tZ−∞∞
malization
methods over 8 different channels taken from four
−z
1
(14)
=
fX (t)fY () dt
different layers: the 1st, 3rd, 7th intermediate convolutionalt−∞ |t|
layers and the last classification layer. Fig. 7 shows the= fZ (−z).
comparison between baseline and other normalization methods
Therefore, the distribution of product random variable Z including WA. Fig. 8 shows the cases when IN, LN and GN
will be symmetric around zero, if the distribution of one of are used in conjunction with our WA.
independent random variable X and Y is symmetric around
C. Visualizations of weights and activations in Section IV-Czero.
We visualize the distributions of weights and activations
VII. E XPLANATION OF WA EQUATION FORM
as epoch increases during training. Fig. 9 and Fig. 10 show
In section III-B, we give out the expression for WA as weights distributions of a single channel in a convolutional
Eq. (10). Compared with BN, WA does not have the β layer. The weights distributions of WA and BN are smooth andterm since it manipulates the weight directly instead of the symmetric around zero, while the ones of other normalization
methods are rough or asymmetric. Adding WA with LN, IN
activations. If we add a β term to Eq. (10), it will result in,
and GN smooths and symmetrizes the weights distributions
w − E[w]ŵ = γ p+ β.
(15) of channels. Fig. 11 and Fig. 12 show activation distribun/2 · Var[w]tions of a single channel in a convolutional layer. Similarly,
When it multiplies with the input activations x, we will get an the activation distributions of WA and BN are smooth andextra βx, which is similar to the element in residual blocks. symmetric around zero, while the ones of other normalization
This will introduce the drift of activation variance again, which methods are rough or asymmetric. Adding WA with LN, IN
and GN smooths and symmetrizes the activation distributions
goes against our original intention.
of channels.
VIII. A DDITIONAL E XPERIMENTS AND V ISUALIZATION
A. Scale factor in WeightAlign
We here conduct an ablationp
study experiments to validate
the effect of our scale factor n/2 in Eq. (10). Specially,
we compare our scale factor with a 0.2x our scale factor, 2x
our scale factor and 4x our scale factor. The experiments
arepshown in Fig. 6. We find that a similar scale to n/2 will

(a) Baseline(b) WA(c) BN(d) LN(e) IN(f) GN
Fig. 7. Each color represents the activation distribution of different channels for different layers. The first three columns denote 1st, 3rd, 7th convolutional
layers and the last one presents the last classification layer before softmax. All normalization methods can reduce internal covariate shift to some extent
comparing with baseline.

(a) BN+WA(b) LN+WA(c) IN+WA(d) GN+WA
Fig. 8. The proposed WA can be used in conjunction with BN, LN, IN and GN. The first three columns denote 1st, 3rd, 7th convolutional layers and thelast one presents the last classification layer before softmax. Note that the activation we plot here is before passing through specific normalization layer.

(a) Baseline(b) WA(c) BN(d) LN(e) IN(f) GN

Fig. 9. Weights distributions of a single channel in a convolutional layer for different normalization methods. WA and BN have smooth and symmetric weight
distributions.

(a) WA + BN(b) WA + LN(c) WA + IN(d) WA + GN

Fig. 10. Weights distributions of a single channel in a convolutional layer for different normalization methods in conjunction with WA method. Adding WA
smooths and symmetrizes the weight distributions.

(a) Baseline(b) WA(c) BN(d) LN(e) IN(f) GN

Fig. 11. Activation distributions of a single channel in a convolutional layer for different normalization methods. WA and BN have smooth and symmetricweight distributions.

(a) WA + BN(b) WA + LN(c) WA + IN(d) WA + GN

Fig. 12. Activation distributions of a single channel in a convolutional layer for different normalization methods in conjunction with WA method. Adding
WA smooths and symmetrizes the activation distributions.