Any purpose to set bias=False in DenseNet torchvision?

John1231983: <p>Hello all, I have read implementation of densenet in torch vision example. I found that they set bias=False in convolution after batch norm layer. It looks redundance because the batch norm did it. Do we have any reason why they add it? Thanks</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/b51a2c316cf80b8cb5f708dab51fbfa0f83f73d2/torchvision/models/densenet.py#L132" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/b51a2c316cf80b8cb5f708dab51fbfa0f83f73d2/torchvision/models/densenet.py#L132" target="_blank" rel="nofollow noopener">pytorch/vision/blob/b51a2c316cf80b8cb5f708dab51fbfa0f83f73d2/torchvision/models/densenet.py#L132</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="122" style="counter-reset: li-counter 121 ;">
<li>            del state_dict[key]</li>
<li>    model.load_state_dict(state_dict)</li>
<li>return model</li>
<li>
</li>
<li>
</li>
<li>class _DenseLayer(nn.Sequential):</li>
<li>def __init__(self, num_input_features, growth_rate, bn_size, drop_rate):</li>
<li>    super(_DenseLayer, self).__init__()</li>
<li>    self.add_module('norm1', nn.BatchNorm2d(num_input_features)),</li>
<li>    self.add_module('relu1', nn.ReLU(inplace=True)),</li>
<li class="selected">    self.add_module('conv1', nn.Conv2d(num_input_features, bn_size *</li>
<li>                    growth_rate, kernel_size=1, stride=1, bias=False)),</li>
<li>    self.add_module('norm2', nn.BatchNorm2d(bn_size * growth_rate)),</li>
<li>    self.add_module('relu2', nn.ReLU(inplace=True)),</li>
<li>    self.add_module('conv2', nn.Conv2d(bn_size * growth_rate, growth_rate,</li>
<li>                    kernel_size=3, stride=1, padding=1, bias=False)),</li>
<li>    self.drop_rate = drop_rate</li>
<li>
</li>
<li>def forward(self, x):</li>
<li>    new_features = super(_DenseLayer, self).forward(x)</li>
<li>    if self.drop_rate &gt; 0:</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Could you explain your concern a bit?</p>
<p>Usually the <code>bias</code> is removed in conv layers before a batch norm layer, as the batch norm’s <code>beta</code> parameter (<code>bias</code> of <code>nn.BatchNorm</code>) will have the same effect and the <code>bias</code> of the conv layer might be canceled out by the mean subtraction.</p>
<p>From the <a href="https://arxiv.org/pdf/1502.03167.pdf">batch norm paper</a>:</p>
<blockquote>
<p>Note that, since we normalize Wu+b, the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction (the role of the bias is subsumed by β in Alg. 1).</p>
</blockquote>

John1231983: <p>Sorry i mistaken. Bias= false means does not set bias and default is True. Forget it. Thanks so much</p>

pvardanis: <p>I guess if we add <code>bias=True</code> on layers after batch_norm layers then there shouldn’t be any issue right? At least that’s what I understand from the last layers of  <code>torchvision.models.vgg16_bn()</code>:</p>
<blockquote>
<p>(avgpool): AdaptiveAvgPool2d(output_size=(7, 7))<br>
(classifier): Sequential(<br>
(0): Linear(in_features=25088, out_features=4096, bias=True)<br>
(1): ReLU(inplace=True)<br>
(2): Dropout(p=0.5, inplace=False)<br>
(3): Linear(in_features=4096, out_features=4096, bias=True)<br>
(4): Dropout(p=0.5, inplace=False)<br>
(5): Linear(in_features=4096, out_features=1024, bias=True)<br>
)</p>
</blockquote>

ptrblck: <p>There shouldn’t be any error regarding the code. However, as mentioned above you might save the computation in specific use cases.</p>