`BatchNorm1d()` with batchsize=1

hcl: <p>Q1: How does <code>BatchNorm1d()</code> judge the current <code>forward()</code> is training or inference? Is there some parameters can be observed and setted manually?<br>
Q2: Specifically speaking, I’m trying to implement a reinforcement learning task. While in the training process, the transition tuple &lt;state, action, state_next, reward&gt; has to be generated one by one calling <code>forward()</code>, the state is input and the reward and the action are outputs, and then is add to the experience reply memory for the subsequent minibatch optimization, which means in the process of generating the transition tuple, the batch size of the input of <code>forward()</code> is 1.  Hence, the <code>BatchNorm1d()</code> can not work because the standard deviation is 0. How to deal with the problem effectively? My existing simple idea is to randomly generate some transition tuple not using <code>forward()</code> for initialization. Is there some effectively and professional methods?<br>
Thanks.</p>

ptrblck: <ol>
<li>
<p>The internal <code>.training</code> attribute determines the behavior of some layers, e.g. batch norm layers.<br>
If you call <code>model.train()</code> or <code>model.eval()</code> (or <code>model.bn_layer.train()</code>), this internal flag will be switched.</p>
</li>
<li>
<p>If you are using a single sample as your batch, you might consider using other normalization lazers, e.g. <code>InstanceNorm</code>.</p>
</li>
</ol>

nono: <p>Hi there,</p>
<p>I run on multiple GPUs with random varying batch sizes.<br>
I found that sometimes the last batch of the epoch is of size 1 and than BatchNorm1D throws an error and stops my run.</p>
<p>I solved locally with try&lt;-&gt;except but it seems like a bug to me…</p>

ptrblck: <p>To remove the last, potentially smaller, batch from an epoch, you could specify <code>drop_last = True</code> in your <code>DataLoader</code>.</p>

kontrabas: <p>How can I use InstanceNorm layer with samples (batch size = 1)?</p>
<p>When I’m trying to do something like this:</p>
<pre><code class="lang-auto">x = torch.randn([1, 512])
m = nn.InstanceNorm1d(512)
m(x)
</code></pre>
<p>I’m getting error like this:</p>
<pre><code class="lang-auto">InstanceNorm1d returns 0-filled tensor to 2D tensor.This is because InstanceNorm1d reshapes inputs to(1, N * C, ...) from (N, C,...) and this makesvariances 0.
</code></pre>
<p>Did I misunderstand something?</p>

ptrblck: <p><code>nn.InstanceNorm1d</code> will calculate the statistics for each sample in the batch separately.<br>
While this might be an advantage over batchnorm layers for small batch sizes (unsure if <code>InstanceNorm</code> would perform better than e.g. <code>GroupNorm</code>), the statistics would still need more than a single scalar in the temporal dimension.</p>

Brando_Miranda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I know this might be a strange question but its related. I want to remember the previously used batch statistics (without using a running average) then evaluate on 1. Is that possible or is the only way to create my own batch norm layer?</p>
<p>see: <a href="https://discuss.pytorch.org/t/how-to-use-have-batch-norm-not-forget-batch-statistics-it-just-used/103437" class="inline-onebox">How to use have batch norm not forget batch statistics it just used?</a></p>

FusionCarcass: <p>In the model I am training, I can only submit batch sizes of one through the model during training due to memory constraints. Each of my input tensors are (1, 1, 192-65536). I could normalize to (1, 1, 65536) so that I could do batching, but that means that in my Conv1d(512, 768, 3) layer, that the memory required to house all of those dimensions with a batch size of 128 would be 128 * 65536 * 4 (bytes per float) * 768 (channels) = 24GB of RAM, but my GTX 1080 only has 8GB of RAM.</p>
<p>Bottom line, I can’t push more than one sample per batch. I currently call loss.backward() on each loss for 128 samples which aggregates the gradients. I then call optimizer.step() after all 128 samples have been processed.</p>
<p>I want to include batch normalization. Is there someway to use BatchNorm1d in this scenario, or would it be possible to implement BatchNorm1d with just the gradients produced by calling loss.backward() 128 times?</p>