Running mean/variance, biased or unbiased, running var via running mean or cur mean

AlbertZeyer: <p>Currently for batch norm:</p>
<ul>
<li>Batch mean and biased batch variance is used during training for renormalization.
<ul>
<li>Why biased batch variance and not unbiased batch variance?</li>
<li>Why not running mean/var? I would guess the running mean/var are better, esp if the mini batches are small?</li>
</ul>
</li>
<li>Batch mean and unbiased batch variance is used during training to update running mean/var.
<ul>
<li>Why unbiased variance here instead of biased variance?</li>
</ul>
</li>
<li>Batch variance is estimated using the batch mean.
<ul>
<li>Why using the batch mean and not the running mean? The running mean should be a better estimate of the mean than the batch mean?
<ul>
<li>If using the running mean, is this always unbiased?</li>
</ul>
</li>
</ul>
</li>
</ul>
<hr>
<p>I found some related discussions (but still don’t fully get the conclusions from those, i.e. what would be the answers to my questions above):</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/issues/1410" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm should use Bessel's correction consistently · Issue #1410 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/77427" class="inline-onebox" rel="noopener nofollow ugc">Batch norm doc mentions using biased variance estimator, but it actually uses the unbiased variance estimator · Issue #77427 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://datascience.stackexchange.com/questions/24239/batch-normalization-variance-calculation" class="inline-onebox" rel="noopener nofollow ugc">deep learning - Batch normalization variance calculation - Data Science Stack Exchange</a></li>
<li><a href="https://stats.stackexchange.com/questions/311074/batch-normalization-variance-calculation/" class="inline-onebox" rel="noopener nofollow ugc">deep learning - Batch normalization variance calculation - Cross Validated</a></li>
</ul>

sally2: <p>Hey there,<br>
I hope my response clarify your doubt:</p>
<ul>
<li>Biased batch variance is used during training for renormalization because it’s faster to compute. However, using running mean/variance, especially with small mini-batches, might yield better results.</li>
<li>Unbiased batch variance is used during training to update running mean/var because it provides a more accurate estimation.</li>
<li>Batch mean is used to estimate batch variance because it’s readily available. However, using the running mean could potentially yield better estimates of the mean.</li>
</ul>
<p>Using the running mean doesn’t always guarantee unbiasedness; it depends on the specific implementation.</p>

AlbertZeyer: <p>Thanks for the reply!</p>
<aside class="quote no-group" data-username="sally2" data-post="2" data-topic="204486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sally2/48/70596_2.png" class="avatar"> sally2:</div>
<blockquote>
<p>Biased batch variance is used during training for renormalization because it’s faster to compute.</p>
</blockquote>
</aside>
<p>Faster than what? Faster than running variance? But you anyway compute the running var, so I don’t understand where you would save sth? Or faster than unbiased batch variance? But I don’t understand how there can be any difference? In one case, after taking the sum (<code>var_sum = sum((x - mean)**2)</code> with <code>mean = sum(x)/n</code>), you divide by n (<code>var_sum/n</code>), in the other case, you divide by (n-1) (<code>var_sum/(n-1)</code>). See code <a href="https://github.com/pytorch/pytorch/blob/59605811488eb07b3b8bf70a5f0b4b56b34b4a61/aten/src/ATen/native/Normalization.cpp#L263" rel="noopener nofollow ugc">here</a>.</p>
<aside class="quote no-group" data-username="sally2" data-post="2" data-topic="204486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sally2/48/70596_2.png" class="avatar"> sally2:</div>
<blockquote>
<p>Unbiased batch variance is used during training to update running mean/var because it provides a more accurate estimation.</p>
</blockquote>
</aside>
<p>But then why is the unbiased batch variance not used during training for renormalization?</p>
<aside class="quote no-group" data-username="sally2" data-post="2" data-topic="204486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sally2/48/70596_2.png" class="avatar"> sally2:</div>
<blockquote>
<p>Batch mean is used to estimate batch variance because it’s readily available. However, using the running mean could potentially yield better estimates of the mean.</p>
</blockquote>
</aside>
<p>But you also have the running mean readily available?</p>
<aside class="quote no-group" data-username="sally2" data-post="2" data-topic="204486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sally2/48/70596_2.png" class="avatar"> sally2:</div>
<blockquote>
<p>Using the running mean doesn’t always guarantee unbiasedness; it depends on the specific implementation.</p>
</blockquote>
</aside>
<p>I mean <code>sum((x - running_mean)**2) / n</code>. Is this biased or unbiased?</p>