Error: Expected more than 1 value per channel when training

AlexisW: <p>I have a model that works perfectly when there are multiple input. However, if there is only one datapoint as input, I will get the above error. Does anyone have an idea on what’s going on here?</p>

ptrblck: <p>Most likely you have a <code>nn.BatchNorm</code> layer somewhere in your model, which expects more then 1 value to calculate the running mean and std of the current batch.<br>
In case you want to validate your data, call <code>model.eval()</code> before feeding the data, as this will change the behavior of the <code>BatchNorm</code> layer to use the running estimates instead of calculating them for the current batch.<br>
If you want to train your model and can’t use a bigger batch size, you could switch e.g. to <code>InstanceNorm</code>.</p>

AlexisW: <p>You are right - adding <code>model.eval()</code> is the key.</p>

Ahsan_Habib: <p>Consider the following network snippet</p>
<pre><code class="lang-auto">def __init__(self):
  ...
  self.fc_hidden_0 = nn.Linear(586, 100)
  self.bn_hidden_0 = nn.BatchNorm1d(100)

def forward(self, X):
  ...
  out = F.leaky_relu(self.bn_hidden_0(self.fc_hidden_0(out)))
</code></pre>
<p>Batch size is 32 (tried 20 as well) and sometime the the error is thrown -</p>
<pre><code class="lang-auto">ValueError: Expected more than 1 value per channel when training, got input size [1, 100]
</code></pre>
<p>I was trying to understand <span class="hashtag">#ptrblck</span> response that BatcNorm could be the problem which expects 100 values but does it expect that value of batch size  32 x 100 and getting a single instance as may be the train-loader is getting the last data segment?</p>
<p>Note that when I restart the training (followed by validation and train again) then the error is gone. Also note that model train() and eval() is placed before the starting of training and validation.</p>

ptrblck: <p>The error is most likely thrown during training and if the current batch only contains a single sample.<br>
As you’ve explained this might happen, if the length of your <code>Dataset</code> is not dividable by the batch size without a remainder, which happens to be 1.<br>
You could set <code>drop_last=True</code> in your <code>DataLoader</code> and run your code again.</p>

Mandy: <p>I met the same problem. I’m using the <code>Deeplabv3</code> from the <code>model.zoo</code>, and I only like to train one image, so my batch size is 1, my input image size is <code>512*1024</code>,and I got the exactly same error.</p>
<pre><code class="lang-python">Traceback (most recent call last):
  File "/home/mengdietao/.pycharm_helpers/pydev/pydevd.py", line 1758, in &lt;module&gt;
    main()
  File "/home/mengdietao/.pycharm_helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/mengdietao/.pycharm_helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/mengdietao/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/mengdietao/tangshan/tangshan_data/image_segmentation/docs/data_check.py", line 356, in &lt;module&gt;
    train(CONFIG, True)
  File "/home/mengdietao/tangshan/tangshan_data/image_segmentation/docs/data_check.py", line 244, in train
    pred = model(data)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torchvision/models/segmentation/_utils.py", line 22, in forward
    x = self.classifier(x)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torchvision/models/segmentation/deeplabv3.py", line 91, in forward
    res.append(conv(x))
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torchvision/models/segmentation/deeplabv3.py", line 60, in forward
    x = super(ASPPPooling, self).forward(x)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/home/mengdietao/.conda/envs/mengdietao/lib/python3.7/site-packages/torch/nn/functional.py", line 1652, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])
</code></pre>

Wendell_Philips: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
But why when i use F.interploate with batchsize=1, i meet the same error ?</p>

ptrblck: <p><code>F.interpolate</code> should not throw this error, as it’s underlying operations are independent regarding the batch size.<br>
Could you check your model for batchnorm layers?<br>
If you don’t find anything suspicious, could you post a code snippet to reproduce this error?</p>

Wendell_Philips: <pre><code class="lang-auto"> x5 = F.interpolate(self.avg_pool(x), size=x.size()[2:], mode='bilinear', align_corners=True)
</code></pre>
<p>That’ a BN layer in avg_pool. Thanks very much!</p>

hy_struggle: <p>thanks, the func <code>model.eval()</code> is useful.</p>

Zhengfang_Xin: <p>I want to train the model and <code>batch_size = 1</code>, so I add <code>drop_last = True</code> to my training dataloader. However, the error is still there.<br>
And the error doesn’t happen in the backbone. It existed at deeplabV3-assp-<code>x5 = F.interpolate(self.avg_pool(x), size=x.size()[2:], mode='bilinear', align_corners=True)</code></p>

ptrblck: <p>For a batch size of 1 you don’t need to use <code>drop_last=True</code>, as there won’t be a smaller batch at the end of the epoch.</p>
<p>The error should be raised by batchnorm layers, which cannot calculate the batch statistics using a single sample.<br>
You should therefore either increase the batch size or call <code>eval()</code> on the batchnorm layers.</p>

Alpha: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I use the <code>nn.SyncBatchNorm</code>  with bath size of 1 for each of the 2 GPUs.</p>
<p>I.e, the total batch size is 2.  But I still get the same error.</p>
<p>I use the pytorch1.5.</p>

ptrblck: <p>Each device would still need more than a single value to calculate the local estimates.<br>
<code>SyncBatchNorm</code> will only synchronize the calculations between the devices.</p>

Alpha: <p>Thank you very much!</p>

Alpha: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>One strange problem: when I use pytorch1.3.0,  I use the  <code>nn.SyncBatchNorm</code>  with bath size of 1 for each of the 8 GPUs. I.e. The total batch size is 8. I didn’t get this error. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

ptrblck: <p>Could you check the shape of the input activation to this particular batchnorm layer, please?</p>

Alpha: <p>When the shape of the input to the <code>nnSyncBatchNorm</code> is <code>(1, 128, 32, 32)</code> for each gpu, it is OK to use the <code>SyncBatchNorm</code> if I have more than a sinle GPU.</p>
<p>But, when the input shape is <code>(1, 128, 1, 1)</code>, i.e, <code>1x1</code> spatial resolution,   <strong>each device would still need more than a single value to calculate the local estimates</strong>， as mentioned by  <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>This phenomenon is based on pytorch1.5.</p>
<p>And when on pytorch1.3, <code>nn.SyncBatchNorm</code> always works regardless of the spatial resolution of input.</p>
<p>Is that a  problem for PyTorch? 1.3 or 1.5? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks in advance.</p>

ptrblck: <p>I’m not sure, if <code>1.3</code> had a bug, but this functionality should have been added approx. a month agp in <a href="https://github.com/pytorch/pytorch/pull/37133">this PR</a>, which would enable it in the nightly binaries and next release. Could you install the nightlies and check, if it’s working now?</p>

Alpha: <p>sorry for late reply.</p>
<p>I’m stucked by the <a href="https://github.com/pytorch/pytorch/issues/39101" rel="nofollow noopener">issue</a> when install pytorch from source code</p>