Error when resizing batch

slavavs: <p>I coached a model with batch = 500. I saved the training data and want to download from the model for training with batch = 100, but I get an error from batchNorm1d.<br>
Can I get around this error so as not to lose my stored data during a workout?</p>

slavavs: <p>It is difficult for me to ask the right question, but when they understand me, I get a good answer.<br>
I cannot continue to explore the network until I find the answer to my question.<br>
From the beginning, I coached the network for one feature per cycle. This happened a very long time. Now I submit to the network a batch of features. It works well. But I need to test the network by serving one feature at a time. But when I load load_state_dict and try to submit features one by one, I get an error. Since during training, nn.BatchNorm1d took the size of the party, and during the test I submit a size one. How do I solve this problem?</p>

slavavs: <p>I used bn1d, it gave me an error. But I increased the dimension to 4D and used bn2d. The error has gone.</p>

ptrblck: <p>Call <code>model.eval()</code> before passing single samples to the model.<br>
This will make sure to use the running stats in the batch norm layers and should avoid this error.</p>
<p>PS: don’t forget to add a batch dimension even to a single sample.</p>

slavavs: <p>I tried two options, but I don’t know if they behave the same way.</p>
<ol>
<li>out = out.contiguous (). View (1, batch, 1, wn1) -&gt; self.bn1 = nn.BatchNorm2d (batch, affine = True)</li>
<li>out = out.contiguous (). View (1,1, batch, wn1) -&gt; self.bn1 = nn.BatchNorm2d (batch, affine = True)<br>
In the first case, BatchNorm2d takes batch and does the normalization for each example. In the second case, BatchNorm2d takes 1 and makes normalization for all examples. The second case allows me to change batch without problems. Training in the first case and in the second is the same.</li>
</ol>

ptrblck: <p>The second example should fail, since the specified channels in <code>bn1</code> (<code>batch</code>) do not equal the passed channels (<code>1</code>).</p>
<p>If you are dealing with a temporal signal in the shape <code>[batch_size, channels, sequence_length]</code>, I would stick to <code>nn.BatchNorm1d</code> and just call <code>model.eval()</code> for the test/validation case.</p>
<p>Reshaping your data such that the batch size is in dim1, is most likely not what you want.</p>

slavavs: <aside class="quote no-group quote-modified" data-username="slavavs" data-post="5" data-topic="49819">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slavavs/48/15429_2.png" class="avatar"> slavavs:</div>
<blockquote>
<p>out = out.contiguous (). View (1,1, batch, wn1) → self.bn1 = nn.BatchNorm2d (batch, affine = True)</p>
</blockquote>
</aside>
<p>sorry<br>
2. out = out.contiguous (). View (1,1, batch, wn1) → self.bn1 = nn.BatchNorm2d (1, affine = True)</p>

ptrblck: <p>That would be syntactically correct.<br>
However, I still think you shouldn’t reshape the tensor to get the batch size in dim1 in your current implementation.</p>
<p>Is <code>nn.BatchNorm1d</code> and <code>model.eval()</code> not working?<br>
If so, could you post the stack trace so that we could have a look?</p>

slavavs: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="49819">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>PS: don’t forget to add a batch dimension even to a single sample.</p>
</blockquote>
</aside>
<p>Help understand this solution. How can I add a batch size if I have one sample?<br>
Do I need to add the batch size with which I train the model or set the size to 1?</p>

ptrblck: <p>For a single sample you should set the batch size to 1 using <code>data = data.unsqueeze(0)</code>.</p>