Torch version affects the network's training performance

username: DeH40_Sung
cooked: <p>I am opening this issue because apparently depending on which version of pytorch you are using, the training result will be different. Here are the 3px error evaluation curves of on a minimal example of overfitting the network on a single image for 300 epochs:</p>
<p><a href="https://user-images.githubusercontent.com/13513172/102947407-4136f700-4491-11eb-98da-0bd2e821bb5b.png" rel="noopener nofollow ugc"><img src="https://user-images.githubusercontent.com/13513172/102947407-4136f700-4491-11eb-98da-0bd2e821bb5b.png" alt="Screenshot from 2020-12-22 20-06-19" width="341" height="249"></a></p>
<p>The purple line is trained with <strong>Pytorch 1.7.0</strong> and the orange line is trained with <strong>Pytorch 1.5.1</strong>. As you can see, with <strong>version 1.7.0</strong> the error rate is flat 100%, while <strong>version 1.5.1</strong> the error rate is dropping. Reason for this is that the BatchNorm function has changed between <strong>version 1.5.1</strong> and <strong>Pytorch 1.7.0</strong>. In <strong>version 1.5.1</strong>, if I disable <code>track_running_stats</code> <a href="https://github.com/mli0603/stereo-transformer/blob/main/module/sttr.py#L52" rel="noopener nofollow ugc">here</a>, both evaluation and training will use batch stats. However in <strong>Pytorch 1.7.0</strong>, it is forced to use <code>running_mean</code> and <code>running_var</code> in evaluation mode, while in training the batch stats is used. With <code>track_running_stats</code> disabled, the <code>running_mean</code> is 0 and <code>running_var</code> is 1, which is clearly different from the batch stats.</p>
<p>How can I maintain this feature beyond version 1.5（evaluation using batch stats）？</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="DeH40_Sung" data-post="1" data-topic="141642">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deh40_sung/48/45764_2.png" class="avatar"> DeH40_Sung:</div>
<blockquote>
<p>In <strong>version 1.5.1</strong>, if I disable <code>track_running_stats</code> <a href="https://github.com/mli0603/stereo-transformer/blob/main/module/sttr.py#L52">here</a>, both evaluation and training will use batch stats. However in <strong>Pytorch 1.7.0</strong>, it is forced to use <code>running_mean</code> and <code>running_var</code> in evaluation mode, while in training the batch stats is used. With <code>track_running_stats</code> disabled, the <code>running_mean</code> is 0 and <code>running_var</code> is 1, which is clearly different from the batch stats.</p>
</blockquote>
</aside>
<p>I don’t think that’s true, as seen in the code for <a href="https://github.com/pytorch/pytorch/blob/v1.5.0/torch/nn/modules/batchnorm.py#L31-L38">1.5.0</a> vs. <a href="https://github.com/pytorch/pytorch/blob/v1.7.0/torch/nn/modules/batchnorm.py#L45-L52">1.7.0</a>. As you can see, in both versions the running stats are initialized as <code>None</code>.<br>
Also, this code snippet shows the same behavior:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

print(torch.__version__)

bn = nn.BatchNorm2d(3, track_running_stats=False)
print(bn.running_mean, bn.running_var)
x = torch.randn(2, 3, 24, 24) * 5 + 2

out = bn(x)
print(out.min(), out.max(), out.mean(), out.std())

bn.eval()
out = bn(x)
print(out.min(), out.max(), out.mean(), out.std())
</code></pre>
<p>Output:</p>
<pre data-code-wrap="python"><code class="lang-python">1.5.0+cpu
None None
tensor(-3.5741, grad_fn=&lt;MinBackward1&gt;) tensor(3.5072, grad_fn=&lt;MaxBackward1&gt;) tensor(1.6557e-09, grad_fn=&lt;MeanBackward0&gt;) tensor(1.0001, grad_fn=&lt;StdBackward0&gt;)
tensor(-3.5741, grad_fn=&lt;MinBackward1&gt;) tensor(3.5072, grad_fn=&lt;MaxBackward1&gt;) tensor(1.6557e-09, grad_fn=&lt;MeanBackward0&gt;) tensor(1.0001, grad_fn=&lt;StdBackward0&gt;)

1.7.0+cpu
None None
tensor(-3.3744, grad_fn=&lt;MinBackward1&gt;) tensor(3.2699, grad_fn=&lt;MaxBackward1&gt;) tensor(3.8633e-09, grad_fn=&lt;MeanBackward0&gt;) tensor(1.0001, grad_fn=&lt;StdBackward0&gt;)
tensor(-3.3744, grad_fn=&lt;MinBackward1&gt;) tensor(3.2699, grad_fn=&lt;MaxBackward1&gt;) tensor(3.8633e-09, grad_fn=&lt;MeanBackward0&gt;) tensor(1.0001, grad_fn=&lt;StdBackward0&gt;)
</code></pre>
<p>so the difference might be coming from another part of the code.</p>