ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])

Dhruv: <p>I have a four gpu setup (24 GB each) where I am trying to train a DeepLabV3Plus model using <a href="https://github.com/qubvel/segmentation_models.pytorch" rel="noopener nofollow ugc">segmentation_models_pytorch</a> library.  <br><br>
I am facing these errors:<br>
<code>ValueError: Caught ValueError in replica 0 on device 0.</code><br>
<code>ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])</code></p>
<p>The batch size I am using is 8.<br>
Can you please help me resolve this issue?<br>
Thank you!</p>

eqy: <p>It looks like you are passing a grayscale image (single channel) when the model expects multiple channels.</p>
<p>Are you adjusting for this property when creating model e.g., via the <code>in_channels</code> argument?</p>
<pre><code class="lang-auto">model = smp.Unet(
...
    in_channels=1,                  # model input channels (1 for gray-scale images, 3 for RGB, etc.)
...
)
</code></pre>

Dhruv: <p>Actually, I am using all RGB images.</p>

Dhruv: <p>It works with the default settings. Now there is a pool of encoders among which we can select which one to use. I am experimenting with different encoders. For some it works, for some it doesn’t.</p>
<p>Thank you for replying so quickly!<br>
Really appreciate it.</p>

eqy: <p>Right, I misread the original error. It looks like your “encoder” might be downsampling the input too much. Could you check if e.g., downsampling is a setting that is available to you or workaround the issue by increasing the input size so that the spatial dimensions are not reduced to 1x1?</p>

Dhruv: <p><a href="https://smp.readthedocs.io/en/latest/models.html#id9" rel="noopener nofollow ugc">These</a> are all the available settings.<br>
And currently I am using everything default except for the <code>encoder_name</code>.</p>

eqy: <p>Right, so you might look into:<br>
decreasing <code>encoder_output_stride</code>, increasing <code>upsampling</code>, or increasing the resolution of the input image as possible solutions.</p>

Dhruv: <p>Okay. Thank you for the tips. I am gonna try these out, and will update on this thread!<br>
Thank you so much!</p>

ptrblck: <p><a class="mention" href="/u/eqy">@eqy</a> already explained why the error might be raised, however it still doesn’t fit your description:</p>
<blockquote>
<p>I have a four gpu setup (24 GB each) where I am trying to train a DeepLabV3Plus<br>
<code>got input size torch.Size([1, 256, 1, 1])</code></p>
</blockquote>
<p>I don’t know of you are using data parallel (I would assume so), which should yield a batch size of 2 for each of the 4 GPUs assuming the global batch size is 8. If the local batch size is set to 8 then of course each GPU should get 8 samples while the error indicates a single sample.</p>

Dhruv: <p>Yes, I am using <code>DataParallel</code>.</p>

ptrblck: <p>Which would mean that each of the four GPUs should process 2 samples for a batch size of 8. Could you add <code>print</code> statements to the <code> forward</code> method and post the shape of the input as well as all activation tensors? I guess you might either use an invalid reshaping operation in the <code>forward</code> or your batch size is not 8.</p>

QingCheng24: <p>I had the same issue which was solved by passing <code>drop_last = True</code> to the Dataloader.<br>
The issue occurs when the last batch accidentally holds 1 sample.</p>