Weight decay in the optimizers is a bad idea (especially with BatchNorm)

Michael_Oliver: <p>Correct me if I’m wrong, but there is no reason the beta and gamma parameters in BatchNorm should ever be subject to weight decay, ie L2 regularization, that pulls them toward 0. In fact it seems like a very bad idea to pull them toward 0. I know you can use Per-parameter options to get around the optimizers default behavior, but it seems like bad default behavior, especially with how commonly batchnorm is used. It seems better to only specify regularization where you want it rather than have the default apply it everywhere and thus even places it shouldn’t. I believe Pytorch could actually a cleaner regularization framework, perhaps similar to what we implemented in Keras. Thoughts?</p>

Chuong98: <p>It has been a year, but no feedback on it ?<br>
I absolutely agree with <a class="mention" href="/u/michael_oliver">@Michael_Oliver</a> . For example, in the official example training on ImageNet    <a href="https://github.com/pytorch/examples/tree/master/imagenet" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/imagenet</a>, it seems to me that Batch Norm Weight and Bias are added to Regularization Loss.<br>
I think the correct way to implement it shoulde be:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD( model.parameters(), args.lr,
                                momentum=args.momentum)
                                # ,weight_decay=args.weight_decay) #Remove weight decay in here
cls_loss = criterion(output, target)
reg_loss = 0
for name,param in model.named_parameters():
    if 'bn' not in name:
         reg_loss += torch.norm(param)
loss = cls_loss + args.weight_decay*reg_loss #Manual add weight decay
</code></pre>
<p>Please confirm or help with more elegant solutions. Thank you.</p>

rwightman: <p>I usually create a fn like add_weight_decay below. In current form it will add all batch norm parameters and bias layers to the no_decay list. I use this instead of looking for ‘bn’ strings in the name because that isn’t always consistent from model to model. It’s usually suggested that bias params should also not be decayed, so this does the job for me. You can still use the name if it works for your case though. Separating it otherwise would be a pain.</p>
<pre><code class="lang-auto">def add_weight_decay(model, weight_decay=1e-5, skip_list=()):
    decay = []
    no_decay = []
    for name, param in model.named_parameters():
        if not param.requires_grad:
            continue
        if len(param.shape) == 1 or name in skip_list:
            no_decay.append(param)
        else:
            decay.append(param)
    return [
        {'params': no_decay, 'weight_decay': 0.},
        {'params': decay, 'weight_decay': weight_decay}]
</code></pre>
<p>When I create the optimizer, I put this block in front (usually all this is wrapped in a optim creation factory that also picks the optimizer to create from config or cmd args…</p>
<pre><code class="lang-auto">    weight_decay = args.weight_decay
    if weight_decay and filter_bias_and_bn:
        parameters = add_weight_decay(model, weight_decay)
        weight_decay = 0.
    else:
        parameters = model.parameters()

   if args.opt.lower() == 'sgd':
        optimizer = optim.SGD(
            parameters, lr=args.lr,
            momentum=args.momentum, weight_decay=weight_decay, nesterov=args.nesterov)
   ...
</code></pre>

Chuong98: <aside class="quote no-group" data-username="rwightman" data-post="3" data-topic="16994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/48.png" class="avatar"> rwightman:</div>
<blockquote>
<p>(usually all this is wrapped in a optim creation factory that also picks the optimizer to create from config or cmd args…</p>
</blockquote>
</aside>
<p>Nice. Thank you.<br>
But I am just curious why even the official training code still ignores this fundamental error.</p>

huangbiubiu: <p>Will the weight decay on bn layers affect the performance of the model? Why?</p>

Chuong98: <p>Well, Weight decay basically pulls the norm of paramters to 0. In Batch norm, e.g x_hat = (x -beta)/gamma, you don’t want beta and gamma go to 0. Otherwise, BN is meaningless and erroneous.</p>

fgoldmann: <aside class="quote no-group" data-username="Michael_Oliver" data-post="1" data-topic="16994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_oliver/48/94_2.png" class="avatar"> Michael_Oliver:</div>
<blockquote>
<p>Correct me if I’m wrong, but there is no reason the beta and gamma parameters in BatchNorm should ever be subject to weight decay, ie L2 regularization, that pulls them toward 0. In fact it seems like a very bad idea to pull them toward 0.</p>
</blockquote>
</aside>
<p>This finding is also backed by this publication:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1706.05350">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon-32x32.png" class="site-icon" width="" height="">

      <a href="https://arxiv.org/abs/1706.05350" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/402;"><img src="https://static.arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png" class="thumbnail" width="" height=""></div>

<h3><a href="https://arxiv.org/abs/1706.05350" target="_blank" rel="noopener nofollow ugc">L2 Regularization versus Batch and Weight Normalization</a></h3>

  <p>Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>While I very much appreciate the code snippet provided by <a class="mention" href="/u/rwightman">@rwightman</a> , I too would like to see at least an option in PyTorch to exclude normalization layers from weight decay.</p>

Hzzone: <p>Please refer to the code of optimizer in PyTorch. In detail, after backward, the weight will be added to the grad of weight~(L2 weight decay). We could also directly use the above solution to avoid apply weight decay to bn. However, I have another more elegant method like function below:</p>
<pre><code class="lang-python">
def apply_weight_decay(*modules, weight_decay_factor=0., wo_bn=True):
    '''
    https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/5
    Apply weight decay to pytorch model without BN;
    In pytorch:
        if group['weight_decay'] != 0:
            grad = grad.add(p, alpha=group['weight_decay'])
    p is the param;
    :param modules:
    :param weight_decay_factor:
    :return:
    '''
    for module in modules:
        for m in module.modules():
            if hasattr(m, 'weight'):
                if wo_bn and isinstance(m, torch.nn.modules.batchnorm._BatchNorm):
                    continue
                m.weight.grad += m.weight * weight_decay_factor
</code></pre>
<p>Please note that it should be applied after <code>loss.backward</code>. Besides, as far as I know, all bn layers in pytorch are inherited from  <code>_BatchNorm</code> class.</p>

JIALI_MA: <p>I think you may have mistaken beta and gamma with the running mean and running variance.<br>
The correct bn operation should be:<br>
x_hat = [(x-running_mean)/running_var]*gamma + beta<br>
So the beta and gamma are actually the bias and weight in bn layer.</p>

mMagmer: <p><a class="mention" href="/u/rwightman">@rwightman</a>, i’m no expert, but as <a class="mention" href="/u/jiali_ma">@JIALI_MA</a> suggested, the running averages of batch norm layer are buffers not parameters, hence they can’t be in <code>model.parameters()</code></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
# With Learnable Parameters
m = nn.BatchNorm2d(5)
</code></pre>
<pre><code class="lang-auto">print(dict(m.named_parameters()))
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">{'weight': Parameter containing:
 tensor([1., 1., 1., 1., 1.], requires_grad=True),
 'bias': Parameter containing:
 tensor([0., 0., 0., 0., 0.], requires_grad=True)}
</code></pre>
<pre><code class="lang-auto">print(dict(m.named_buffers()))
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">{'running_mean': tensor([0., 0., 0., 0., 0.]),
 'running_var': tensor([1., 1., 1., 1., 1.]),
 'num_batches_tracked': tensor(0)}
</code></pre>

rwightman: <p>batch norm has learnable affine weight and bias parameters separate from the running mean and var</p>

bilzrd: <p>The following paper[1] shows experimental results that “the effect of regularization was concentrated on the BN layer,” and we should be cautious about making the behavior such that the weight decay of the BN layer is off by default.</p>
<blockquote>
<p>As evidence, we found that almost all of the regularization effect of weight decay was due to<br>
applying it to layers with BN (for which weight decay is meaningless).</p>
</blockquote>
<p>It seems that the mechanism of weight decay is not fully understood even in the research field. At least until there is a clear empirical and theoretical basis, the above modification should be withheld.</p>
<p>[1] <a href="https://arxiv.org/abs/1810.12281" class="inline-onebox" rel="noopener nofollow ugc">[1810.12281] Three Mechanisms of Weight Decay Regularization</a></p>

turian: <p>I would also be interested in a simple option for weight decay not to apply to batchnorm. fast.ai, citing Tencent, <a href="https://www.fast.ai/2018/08/10/fastai-diu-imagenet/" rel="noopener nofollow ugc">reports that removing weight decay for batchnorm was helpful</a>.</p>

addisonklinke: <aside class="quote no-group quote-modified" data-username="rwightman" data-post="3" data-topic="16994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/48.png" class="avatar"> rwightman:</div>
<blockquote>
<p>I use this (i.e. <code>len(param.shape) == 1</code>) instead of looking for ‘bn’ strings in the name because [the name] isn’t always consistent from model to model</p>
</blockquote>
</aside>
<p>The intended logic seems more readable to me when using <code>isinstance()</code>. Here’s an <code>add_weight_decay()</code> variant of <a class="mention" href="/u/hzzone">@Hzzone</a>’s <code>apply_weight_decay()</code></p>
<pre data-code-wrap="python"><code class="lang-python">def add_weight_decay(
        model, 
        weight_decay=1e-5, 
        skip_list=(nn.InstanceNorm, nn.BatchNorm)):
    """Using .modules() with an isinstance() check"""
    decay = []
    no_decay = []
    for module in model.modules():
        params = [p for p in module.parameters() if p.requires_grad]
        if isinstance(module, skip_list):
            no_decay.extend(params)
        else:
            decay.extend(params)
    return [
        {'params': no_decay, 'weight_decay': 0.},
        {'params': decay, 'weight_decay': weight_decay}]
</code></pre>

Equim: <p>However, I found another research stating that weight decay is good for BN in some network architectures.</p>
<blockquote>
<p>Unlike weight decay on weights in e.g. convolutional layers, which typically directly precede nor-<br>
malization layers, weight decay on γ and β can have a regularization effect so long as there is a<br>
path in the network between the layer in question and the ultimate output of the network, as if such<br>
paths do not pass through another normalization layer, then the weight decay is never “undone” by<br>
normalization.<br>
…<br>
Regularization in the form of weight decay on the normalization parameters γ and<br>
β can be applied to any normalization layer, but is only effective in architectures with particular<br>
connectivity properties like ResNets and in tasks for which models are already overfitting.</p>
</blockquote>
<p>in <a href="https://arxiv.org/pdf/1906.03548.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1906.03548.pdf</a></p>

TriKri: <p>What you implement here is not L2 regularization. For L2 regularization, you want to add the squared norm, nor the norm itself. (I have seen that sometimes you also divide the squared norm by 2)</p>
<p>Also, are you sure that the test <code>'bn' not in name</code> is sufficient for determining whether a parameter belongs to a batch normalization layer?</p>
<p>Otherwise, I think your approach looks good.</p>

TriKri: <blockquote>
<p>in <a href="https://arxiv.org/pdf/1906.03548.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1906.03548.pdf </a></p>
</blockquote>
<p>“He et al. (2019) encourage disabling weight decay on γ and β, but ultimately find diminished performance by doing so.” Interesting.</p>
<p>For ResNets, the only parameters that it makes sense to regularize using L2 regularization is essentially the γ parameters of the last batch normalization layer in each block. L2 regularizing any other parameter doesn’t have any effect (as long as you L2 regularize all of them in tandem)—except making the parameters smaller in magnitude and thus more sensitive to change, which actually may mess with the learning rate in a big way—see the paper <a href="https://arxiv.org/abs/2106.15739" rel="noopener nofollow ugc"> On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay</a>.</p>

niceblue: <p>Absolutely : “why even the official training code still ignores this fundamental error”</p>
<p>Essentially this means the (in a lot of cases, requiring any norm in particular) weight decay is completely broken “out of the box” in Pytorch. i.e. you cannot just set a weight decay parameter in the optimiser and expect it to work. I had a hard time even finding this forum when I hit this problem. I was very puzzled why AdamW did not work, and weight decay did not work. Only in a harder-to-find forum like this is it discussed at all.</p>
<p>It would seem very straightforward to allow all norm layers and biases to be weight decay of zero in the optimiser code. It doesn’t even need to be the default setting (as this could break existing code). It can be a additional option as already mentioned here already (4 years ago …)</p>