Different batchsizes give different outputs in model.eval() mode

username: kimjh069
cooked: <p>Hello,<br>
I could not find the solution from anywhere.<br>
Please help me with this problem.</p>
<p>I trained my model with batch size of 32 (with 3 GPUs).<br>
There are <strong>Batchnorm1d</strong>s in the model. ( + some dropouts)</p>
<p>During testing,<br>
I checked</p>
<pre><code class="lang-auto">model.eval()
track_running_stats = False
</code></pre>
<p>When I load a sample test data x, and process with the model, model(x), the result is totally different from the outputs during training.<br>
For example, let’s say the model outputs range 0~0.99 during training with batchsize of 32, while 0~0.05 with batchsize of 1 during testing.</p>
<p>In order to examine further, I loaded 2 or more test data from the dataloader and processed through the model with different batch size and saw different outputs even if the data were the same.</p>
<p>For example, I loaded 4 items (x) from the loader,</p>
<pre><code class="lang-auto">x1 = x[:2] # batchsize 2
x2 = x[2:] # batchsize 2

x11 = torch.cat([x[0], x[0]], axis=0) # batchsize 2 of the same data

y0 = model(x[0:1]) # batchsize 1
y1 = model(x1) # batchsize 2
y2 = model(x2) # batchszie 2
y11 = model(x11) # batchsize 2 of the same data
y = model(x) # batchsize 4

print(torch.allclose(y1, y[:2])) # False. y[:2] is different from y1
print(torch.allclose(y2, y[2:])) # False. y[2:] is also different from y2

print(torch.allclose(y11[0], y11[1])) # True
print(torch.allclose(y1[0], y11[0])) # False. y11[0] outputs the same as y11[1], but different from y1[0]
print(torch.allclose(y1[0], y0[0]), torch.allclose(y11[0], y0[0])) # False, False. y0 is also different from y11[0] or y1[0]


</code></pre>
<p><strong>The problems are</strong></p>
<ol>
<li>The model results in different values according to the batch size during testing.</li>
</ol>
<ul>
<li>y[:2] is different from y1, and y[2:] is also different from y2. y0 is also different from y11[0] or y1[0]</li>
</ul>
<ol start="2">
<li>
<p>Especially, if the batch size is 1 as y0 case, the output histogram ranges 0~0.05<br>
(which is not intended) while case of batchsize 2 or more with different items results in 0~0.99 (which is as intended during training).</p>
</li>
<li>
<p>The model results in the same value if the batchsize is increased manually with the same data. y11[0]==y11[1] returns True. This seems correct but, the histogram still ranges 0~0.05.</p>
</li>
</ol>
<p>I think the problems are due to the Batchnorm1ds.<br>
Can someone help me or give a hint to solve them?</p>

username: SunQpark
cooked: <p>I guess somehow your model is doing something wrong with the batch dimension.<br>
you can narrow your problem down by doing simpler test only on suspicious parts of your model.</p>
<pre><code class="lang-python">def test(model, in_shape):
    batch_size = in_shape[0]
    input_all = torch.randn(*in_shape) # input does not have to be test data
    input_1st_half = input_all[:batch_size//2]
    input_2nd_half = input_all[batch_size//2:]

    output_all = model(input_all)

    output_1st_half = model(input_1st_half)
    output_2nd_half = model(input_2nd_half)

    output_concat = torch.cat([output_1st_half, output_2nd_half], dim=0)
    return torch.allclose(output_all, output_concat)

model = nn.BatchNorm1d(10) # model does not need to be trained
model.eval()

test(model, [16, 10, 31])
</code></pre>

username: ptrblck
cooked: <p>Why are you setting <code>track_running_stats = False</code>?<br>
Based on the docs, it’ll use the batch statistics instead of the running stats, which explains the difference:</p>
<blockquote>
<p><strong>track_running_stats</strong>  – a boolean value that when set to  <code>True</code> , this module tracks the running mean and variance, and when set to  <code>False</code> , this module does not track such statistics and always uses batch statistics in both training and eval modes. Default:  <code>True</code></p>
</blockquote>
<p>Small example:</p>
<pre data-code-wrap="python"><code class="lang-python">bn = nn.BatchNorm2d(3)
bn.eval()

x = torch.randn(2, 3, 24, 24) + 100
out = bn(x)
print((out - x).abs().max())
&gt; tensor(0.0005, grad_fn=&lt;MaxBackward1&gt;)

bn.track_running_stats = False
out = bn(x)
print((out - x).abs().max())
&gt; tensor(100.1011, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

username: kimjh069
cooked: <p>Thank you for advising SunQpark.</p>
<p>I removed all of batchnorm1ds in my model and now training again.<br>
And the problems are gone.<br>
I just wonder what is wrong with using batchnorm1ds…</p>

username: kimjh069
cooked: <p>Thank you for the comments, ptrblck!<br>
I think I found the reason for the problems above.</p>
<p>Originally I was using BatchNorm1ds like in the first code bellow.<br>
With the first code, I had to use <code>track_running_stats = False</code> in order to have the reasonable outputs.<br>
Unless, the outputs were always 0.</p>
<p>The shape of the initial input is (N,400,3): Length 400, channel 3<br>
After nn.Linear layer, it becomes (N,400,32)<br>
Then I wanted to pass it through nn.BatchNorm1d.<br>
I initially thought that the batch norm should be done on 3rd dimension, but as BatchNorm1d takes inputs of shape (N,C,L) I transposed 2nd and 3rd dimension like the code bellow.</p>
<pre><code class="lang-auto"># First code. I had to use track_running_stats = False in order to avoid outputs with all-zero value.
class someModel(torch.nn.Module):
    def __init__(self):
        super(someModel, self).__init__()
        self.input_size = 3 
        self.dims = 32
        self.relu = torch.nn.ReLU()
        self.fc1 = torch.nn.Linear(self.input_size, self.dims)
        self.bn1 = torch.nn.BatchNorm1d(self.dims)

    def forward(self, x):
        # x shape: N,400,3 
        x = self.relu(self.bn1(self.fc1(x).transpose(1,2))).transpose(1,2)
        return x # N,400,32
</code></pre>
<p>After reading your comment, I removed transpose operation and batch norm on 2nd dimension (400) rather than 3rd dimension(32) like code bellow</p>
<pre><code class="lang-auto"># Second code. I do not have to set track_running_stats = False, anymore.
class someModel_2(torch.nn.Module):
    def __init__(self):
        super(someModel_2, self).__init__()
        self.input_size = 3 
        self.dims = 400 # CHANGED HERE 32 -&gt; 400
        self.relu = torch.nn.ReLU()
        self.fc1 = torch.nn.Linear(self.input_size, self.dims)
        self.bn1 = torch.nn.BatchNorm1d(self.dims)

    def forward(self, x):
        # x shape: N,400,3 
        x = self.relu(self.bn1(self.fc1(x))) # removed transpose
        return x # N,400,32
</code></pre>
<p>The problems are now gone and looks second way is right.<br>
And also, don’t need to set <code>track_running_stats = False</code>.</p>
<p><strong>I have one last question though.</strong><br>
I am not familiar with these 3D Tensors (N,C,L) or (N,L,C)…</p>
<p>Why Batch norm should be done on the 2nd dimension of size 400 (L) like in the second way?<br>
Should it not be done on 3rd dimension of size 32 (C ) as it become (N,400,32) after the nn.Linear layer?</p>
<p>Can you or someone explain in easy way?</p>
<p>Thank you once again for fixing my problem, ptrblck!</p>