Batchnorm training vs evaluation

username: asc
cooked: <p>Hi,<br>
First of all sorry for re-addressing this topic,<br>
maybe the problem is that I still don’t understand batchnorms:</p>
<p>Given the following input to my notebook:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17027523e517891d3af2717cb3ace5601f103748.png" data-download-href="https://discuss.pytorch.org/uploads/default/17027523e517891d3af2717cb3ace5601f103748" title="Batchnorm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17027523e517891d3af2717cb3ace5601f103748_2_239x500.png" alt="Batchnorm" data-base62-sha1="3hyfk4U0ZtDZgR5TdFjnt8wXcG4" width="239" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17027523e517891d3af2717cb3ace5601f103748_2_239x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17027523e517891d3af2717cb3ace5601f103748_2_358x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/17027523e517891d3af2717cb3ace5601f103748_2_478x1000.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Batchnorm</span><span class="informations">748×1562 77.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would expect:<br>
First there is a difference between the evaluation and the training,<br>
because the training did not yet have enough iterations to have converged to the mean and variance of the data.</p>
<p>But then after several iterations, which are 100_000 in my case.<br>
Everything should have converged:<br>
and therefore —&gt; training model equals evaluation model (given the same input)</p>
<p>But this is not the case <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>As said before, maybe I don’t understand it.</p>
<p>Thanks in advance for any explanation.</p>

username: eqy
cooked: <p>Note that the batch statistics would be used when batchnorm is in “train” mode, so I suspect that you might need a sufficiently large batch size such as the mean and variance of the batch is very similar to the “converged” value observed during training.</p>

username: asc
cooked: <p>Hi eqy,</p>
<p>How big should I make the batch?</p>
<p>Shouldn’t it be sufficient to run the same batch, very often in order to converge?</p>
<p>This is as far as I understand the purpose of the running mean and running variance.</p>
<p>How would you modify my programm in order to have convergence?<br>
My neural net is basically working in the training mode, because the mean and variance gets automatically calculated from the batch.<br>
But now I would like the to have the neural network work equally as good in the evaluation.<br>
Because the evaluation is the application of the neural network.<br>
But unfortunately now it is using a different mean and variance.<br>
I cant change the batchsize, because this is already all the training data that I got.</p>
<p>This is only a toy example, but with my real neural network and trainning data I have the same problem.<br>
Which makes the neural network a little worse in its final application then it was for training.</p>
<p>Thanks in advance.<br>
asc</p>

username: eqy
cooked: <p>Rereading your question, it seems that you may want to try increasing the value of the momentum parameter (default 0.1), so that convergence (on the same data) can happen faster.</p>
<p>You can also print the values of the <code>running_var</code> and <code>running_mean</code> attributes of the BatchNorm layer to see if they are approaching the empirical var and mean of your data and if/when this stops.</p>

username: asc
cooked: <p>Hi eqy,<br>
I now ran:<br>
“”"<br>
import torch<br>
import torch.nn as nn</p>
<pre><code>    print("Using PyTorch Version %s" %torch.__version__)
    
    in_size  = 100
    n_output = 1
    
    n_input_dim = in_size
    
    n_hidden = 100 
    
    
    net = nn.Sequential(
        
        nn.BatchNorm1d(n_input_dim,affine=True ,momentum = 0.9),
        
        nn.Linear(n_hidden, n_output),
    
    )
    
    torch.manual_seed(0)
    inp = torch.rand(10,100) 
    
    print()
    print("before iteration loop")
    net.train()
    trainA = net(inp)
    trainB = net(inp)
    
    net.eval()
    evaluation = net(inp)
    
    print()
    print("trainB - trainA")
    print(trainB - trainA)
    
    print()
    print("evaluation - trainA")
    print(evaluation - trainA)
    
    
    net.train()
    for i in range(100_000):
        net(inp)
    
    print("running_mean")  
    print( net.state_dict()["0.running_mean"] )
    print("running_var")
    print( net.state_dict()["0.running_var"] )
        
    for i in range(10_000_000):
        net(inp)    
    
    print("running_mean")  
    print( net.state_dict()["0.running_mean"] )
    print("running_var")
    print( net.state_dict()["0.running_var"] )
    
    print("after iteration loop")
    net.train()
    trainA = net(inp)
    trainB = net(inp)
    
    net.eval()
    evaluation = net(inp)
    
    print()
    print("trainB - trainA")
    print(trainB - trainA)
    
    print()
    print("evaluation - trainA")
    print(evaluation - trainA)
    """
</code></pre>
<p>with the result:<br>
“”"<br>
Using PyTorch Version 2.0.1+cpu</p>
<pre><code>    before iteration loop
    
    trainB - trainA
    tensor([[0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.]], grad_fn=&lt;SubBackward0&gt;)
    
    evaluation - trainA
    tensor([[-0.0313],
            [ 0.0118],
            [-0.0051],
            [ 0.0899],
            [ 0.0310],
            [-0.0294],
            [-0.1170],
            [ 0.0252],
            [-0.1052],
            [ 0.0898]], grad_fn=&lt;SubBackward0&gt;)
    running_mean
    tensor([0.5088, 0.6670, 0.3810, 0.5606, 0.4588, 0.4557, 0.5589, 0.5949, 0.4525,
            0.5722, 0.3712, 0.4629, 0.4938, 0.3754, 0.6282, 0.5713, 0.3683, 0.5272,
            0.6635, 0.5736, 0.6476, 0.5023, 0.4490, 0.4922, 0.4125, 0.3985, 0.6730,
            0.2649, 0.4704, 0.3969, 0.5462, 0.6128, 0.4573, 0.5381, 0.4119, 0.4991,
            0.5438, 0.3370, 0.5581, 0.5255, 0.4137, 0.4987, 0.5979, 0.5043, 0.3674,
            0.6004, 0.5504, 0.5365, 0.3826, 0.4306, 0.3937, 0.5267, 0.6587, 0.4738,
            0.5307, 0.4332, 0.4600, 0.5151, 0.3925, 0.3587, 0.3423, 0.5201, 0.4529,
            0.4896, 0.5712, 0.5956, 0.5171, 0.4880, 0.4438, 0.6315, 0.5038, 0.5011,
            0.5118, 0.5845, 0.4569, 0.5135, 0.6359, 0.4807, 0.5716, 0.5164, 0.4200,
            0.3237, 0.4433, 0.6070, 0.5800, 0.6215, 0.4976, 0.4250, 0.5063, 0.6422,
            0.5114, 0.5958, 0.5699, 0.5832, 0.4281, 0.4176, 0.4490, 0.4129, 0.5347,
            0.5866])
    running_var
    tensor([0.1170, 0.0552, 0.1209, 0.1130, 0.0925, 0.0819, 0.0968, 0.1087, 0.0262,
            0.0297, 0.0328, 0.0522, 0.1047, 0.0707, 0.0713, 0.0651, 0.0528, 0.0545,
            0.0546, 0.0703, 0.0448, 0.1295, 0.0633, 0.0918, 0.0917, 0.0963, 0.0891,
            0.0287, 0.0784, 0.0363, 0.1142, 0.1252, 0.1155, 0.0732, 0.1299, 0.0803,
            0.0495, 0.1242, 0.0754, 0.1059, 0.0840, 0.0896, 0.0642, 0.0593, 0.0876,
            0.1371, 0.1110, 0.1040, 0.0339, 0.0738, 0.1149, 0.1198, 0.1107, 0.0914,
            0.0784, 0.0942, 0.0482, 0.1069, 0.0771, 0.0470, 0.0809, 0.0950, 0.1216,
            0.1473, 0.0880, 0.0943, 0.0451, 0.0964, 0.1160, 0.0544, 0.1133, 0.0884,
            0.1439, 0.0576, 0.1036, 0.1378, 0.0962, 0.1441, 0.1185, 0.1162, 0.1056,
            0.1019, 0.1086, 0.0990, 0.0285, 0.0959, 0.0521, 0.0896, 0.0557, 0.0582,
            0.0470, 0.0891, 0.1031, 0.0864, 0.0999, 0.0762, 0.1290, 0.0753, 0.0914,
            0.0715])
    running_mean
    tensor([0.5088, 0.6670, 0.3810, 0.5606, 0.4588, 0.4557, 0.5589, 0.5949, 0.4525,
            0.5722, 0.3712, 0.4629, 0.4938, 0.3754, 0.6282, 0.5713, 0.3683, 0.5272,
            0.6635, 0.5736, 0.6476, 0.5023, 0.4490, 0.4922, 0.4125, 0.3985, 0.6730,
            0.2649, 0.4704, 0.3969, 0.5462, 0.6128, 0.4573, 0.5381, 0.4119, 0.4991,
            0.5438, 0.3370, 0.5581, 0.5255, 0.4137, 0.4987, 0.5979, 0.5043, 0.3674,
            0.6004, 0.5504, 0.5365, 0.3826, 0.4306, 0.3937, 0.5267, 0.6587, 0.4738,
            0.5307, 0.4332, 0.4600, 0.5151, 0.3925, 0.3587, 0.3423, 0.5201, 0.4529,
            0.4896, 0.5712, 0.5956, 0.5171, 0.4880, 0.4438, 0.6315, 0.5038, 0.5011,
            0.5118, 0.5845, 0.4569, 0.5135, 0.6359, 0.4807, 0.5716, 0.5164, 0.4200,
            0.3237, 0.4433, 0.6070, 0.5800, 0.6215, 0.4976, 0.4250, 0.5063, 0.6422,
            0.5114, 0.5958, 0.5699, 0.5832, 0.4281, 0.4176, 0.4490, 0.4129, 0.5347,
            0.5866])
    running_var
    tensor([0.1170, 0.0552, 0.1209, 0.1130, 0.0925, 0.0819, 0.0968, 0.1087, 0.0262,
            0.0297, 0.0328, 0.0522, 0.1047, 0.0707, 0.0713, 0.0651, 0.0528, 0.0545,
            0.0546, 0.0703, 0.0448, 0.1295, 0.0633, 0.0918, 0.0917, 0.0963, 0.0891,
            0.0287, 0.0784, 0.0363, 0.1142, 0.1252, 0.1155, 0.0732, 0.1299, 0.0803,
            0.0495, 0.1242, 0.0754, 0.1059, 0.0840, 0.0896, 0.0642, 0.0593, 0.0876,
            0.1371, 0.1110, 0.1040, 0.0339, 0.0738, 0.1149, 0.1198, 0.1107, 0.0914,
            0.0784, 0.0942, 0.0482, 0.1069, 0.0771, 0.0470, 0.0809, 0.0950, 0.1216,
            0.1473, 0.0880, 0.0943, 0.0451, 0.0964, 0.1160, 0.0544, 0.1133, 0.0884,
            0.1439, 0.0576, 0.1036, 0.1378, 0.0962, 0.1441, 0.1185, 0.1162, 0.1056,
            0.1019, 0.1086, 0.0990, 0.0285, 0.0959, 0.0521, 0.0896, 0.0557, 0.0582,
            0.0470, 0.0891, 0.1031, 0.0864, 0.0999, 0.0762, 0.1290, 0.0753, 0.0914,
            0.0715])
    after iteration loop
    
    trainB - trainA
    tensor([[0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.]], grad_fn=&lt;SubBackward0&gt;)
    
    evaluation - trainA
    tensor([[-0.0052],
            [ 0.0061],
            [-0.0114],
            [ 0.0485],
            [ 0.0164],
            [-0.0136],
            [-0.0431],
            [ 0.0044],
            [-0.0463],
            [ 0.0443]], grad_fn=&lt;SubBackward0&gt;)
    """
</code></pre>
<p>so the evaluation model is still not as good as the training model.</p>
<p>Could you please try to run my program yourself, to see if you can reproduce the issue.<br>
And if so, try to modify it so that it the evaluation model is as good as the training model.</p>
<p>Best Regards,<br>
asc</p>