Why does the resnet model given by pytorch omit biases from the convolutional layer?

barrel-roll: <p>Hi<br>
I was trying to implement my own resnet model by using the model already provided by Pytorch as a reference. I noticed however that none of the convolutional layers had biases, as declared here:</p>
<pre><code>def conv3x3(in_planes, out_planes, stride=1):
    "3x3 convolution with padding"
    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
                                    padding=1, bias=False)
</code></pre>
<p>Is there any reason for this? I didn’t see any mention of removing biases in the original paper.</p>

nikostr: <p>See author’s answer here:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/KaimingHe/deep-residual-networks/issues/10#issuecomment-194037195" target="_blank" rel="nofollow noopener">github.com/KaimingHe/deep-residual-networks</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/liuyipei" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/4404828?v=2&amp;s=96" class="thumbnail onebox-avatar" width="420" height="420">
</a>

<h4><a href="https://github.com/KaimingHe/deep-residual-networks/issues/10#issuecomment-194037195" target="_blank" rel="nofollow noopener">Issue: Should the convolution layers have biases?</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/liuyipei" target="_blank" rel="nofollow noopener">liuyipei</a>
	on <a href="https://github.com/KaimingHe/deep-residual-networks/issues/10#issuecomment-194037195" target="_blank" rel="nofollow noopener">2016-03-09</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/KaimingHe" target="_blank" rel="nofollow noopener">KaimingHe</a>
	on <a href="https://github.com/KaimingHe/deep-residual-networks/issues/10#issuecomment-194037195" target="_blank" rel="nofollow noopener">2016-04-22</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">This is so helpful! Looking at the prototxt, I have a very quick question about whether the convolution layers should have...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The batchnorm layers handle the biases.</p>

barrel-roll: <p>Oh! In that case what are the weights in the batchnorm layer?<br>
I was under the impression that bias and weights in the batchnorm were the expectation and variance respectively.</p>

nikostr: <p>In the original batchnorm paper they mention learnable params beta and gamma:</p>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1502.03167.pdf" target="_blank" rel="nofollow noopener">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1502.03167.pdf" target="_blank" rel="nofollow noopener"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1502.03167.pdf" target="_blank" rel="nofollow noopener">1502.03167.pdf</a></h3>

<p class="filesize">169.48 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>And in pytorch the batchnorm implementation has weights and bias in addition to running mean and running standard deviation.</p>
<p><a href="http://pytorch.org/docs/master/_modules/torch/nn/modules/batchnorm.html" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/_modules/torch/nn/modules/batchnorm.html</a></p>
<p>Edit: The weights and bias are used to scale and shift the output of the layer.</p>

barrel-roll: <p>Okay, I get it now.</p>
<p>Thanks! This clears everything up.</p>

Caroline_Pearl: <p>In the paper section 3.4, last sentence.<br>
Whereas Dropout (Srivastava et al., 2014) is typically used to reduce overfitting,<br>
in a batch-normalized network we found that it can be either removed or reduced in strength.</p>