BatchNorm2d, ValueError: expected 4D input (got 2D input)

MonIA: <p>Hi everyone,<br>
I got ValueError: expected 4D input (got 2D input) when I trained the following network:</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1=nn.Conv2d(3,32,3, padding = 1)<br>
self.conv2=nn.Conv2d(32,64,3, padding = 1)<br>
self.conv2_bn = nn.BatchNorm2d(64)<br>
self.conv3=nn.Conv2d(64,128,3, padding = 1)<br>
self.conv3_bn = nn.BatchNorm2d(128)<br>
self.pool=nn.MaxPool2d(2,2)<br>
self.fc1=nn.Linear(128<em>28</em>28,500)<br>
self.dense1_bn = nn.BatchNorm2d(500)<br>
self.fc2=nn.Linear(500,133)<br>
self.dropout=nn.Dropout(0.20)</p>
<pre><code>def forward(self, x):
   # add sequence of convolutional and max pooling layers
        x=self.pool(F.relu(self.conv1(x)))
        # print(x.shape)
        x=self.pool(F.relu(self.conv2_bn(self.conv2(x))))
        # print(x.shape)
        x=self.pool(F.relu(self.conv3_bn(self.conv3(x))))
        # print(x.shape)
        # flatten image input
        x=x.view(-1,128*28*28)
        x=self.dropout(x)
        x = F.relu(self.dense1_bn(self.fc1(x)))
        # add dropout layer
        x = self.dropout(x)
        x=self.fc2(x)
        return x
</code></pre>
<p>Can you please help me to know why this error happened?</p>

Nikronic: <p>Hi,<br>
The problem is here:</p>
<aside class="quote no-group" data-username="MonIA" data-post="1" data-topic="85110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/monia/48/12827_2.png" class="avatar"> MonIA:</div>
<blockquote>
<p>self.fc1=nn.Linear(128 <em>28</em> 28,500)<br>
self.dense1_bn = nn.BatchNorm2d(500)</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/stable/nn.html#batchnorm2d" rel="noopener nofollow ugc">nn.BatchNorm2d</a> expects 4D inputs in shape of <code>[batch, channel, height, width]</code>. But in the quoted line, you have converted 4D tensor into 2D in shape of <code>[batch, 500]</code> which is not acceptable.</p>
<p>Using <a href="https://pytorch.org/docs/stable/nn.html#batchnorm1d" rel="noopener nofollow ugc">nn.BatchNorm1d</a> will fix the issue.</p>
<pre><code class="lang-auto">self.dense1_bn = nn.BatchNorm1d(500)
</code></pre>
<p>Bests</p>

MonIA: <p>Thank you… the problem is solved! but I have encounter other problem:<br>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 385 and 341 in dimension 2 at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:3586</p>
<p>I’m not sure if this related to the batch size I tried to manipulate it but still the error exists, any tips please?</p>

Nikronic: <p>Could you please share the stacktrace of error and the line it raises?</p>

MonIA: <p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
77 # train the model<br>
78 model_scratch = train(20, loaders_scratch, model_scratch, optimizer_scratch,<br>
—&gt; 79                       criterion_scratch, use_cuda, ‘model_scratch.pt’)<br>
80<br>
81 # load the model that got the best validation accuracy</p>
<p> in train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path)<br>
42         ######################<br>
43         model.eval()<br>
—&gt; 44         for batch_idx, (data, target) in enumerate(loaders[‘valid’]):<br>
45             # move to GPU<br>
46             if use_cuda:</p>
<p>/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in <strong>next</strong>(self)<br>
262         if self.num_workers == 0:  # same-process loading<br>
263             indices = next(self.sample_iter)  # may raise StopIteration<br>
–&gt; 264             batch = self.collate_fn([self.dataset[i] for i in indices])<br>
265             if self.pin_memory:<br>
266                 batch = pin_memory_batch(batch)</p>
<p>/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in default_collate(batch)<br>
136     elif isinstance(batch[0], collections.Sequence):<br>
137         transposed = zip(*batch)<br>
–&gt; 138         return [default_collate(samples) for samples in transposed]<br>
139<br>
140     raise TypeError((error_msg.format(type(batch[0]))))</p>
<p>/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in (.0)<br>
136     elif isinstance(batch[0], collections.Sequence):<br>
137         transposed = zip(*batch)<br>
–&gt; 138         return [default_collate(samples) for samples in transposed]<br>
139<br>
140     raise TypeError((error_msg.format(type(batch[0]))))</p>
<p>/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in default_collate(batch)<br>
113             storage = batch[0].storage().<em>new_shared(numel)<br>
114             out = batch[0].new(storage)<br>
–&gt; 115         return torch.stack(batch, 0, out=out)<br>
116     elif elem_type.<strong>module</strong> == ‘numpy’ and elem_type.<strong>name</strong> != 'str</em>’ <br>
117             and elem_type.<strong>name</strong> != ‘string_’:</p>
<p>RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 385 and 341 in dimension 2 at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:3586</p>

Nikronic: <aside class="quote no-group" data-username="MonIA" data-post="5" data-topic="85110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/monia/48/12827_2.png" class="avatar"> MonIA:</div>
<blockquote>
<p>loaders[‘valid’]</p>
</blockquote>
</aside>
<p>Can you share the way you are constructing <code>loaders</code>? Also, it would be great if you could wrap your codes by a pair of ```. It enhaces readabilty.<br>
Like this:</p>
<pre>```
code
```
</pre>

MonIA: <pre><code class="lang-auto">#number of sub-processes
num_workers = 0
#how many samples per batch to load
batch_size = 40

#transform for the train dataset with aumuntation
transform_trn = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.RandomHorizontalFlip(), #randomly flip and rotate
    transforms.RandomRotation(10),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])
    ])

#transform for validation and testing datasets
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])
transform = transforms.Compose([
           transforms.Resize(256), 
            transforms.ToTensor(),
            normalize,
        ])

#choose the training and test datasets
train_data = datasets.ImageFolder('/data/dog_images/train', transform=transform_trn)
val_data = datasets.ImageFolder('/data/dog_images/valid', transform=transform)
test_data = datasets.ImageFolder('/data/dog_images/test', transform=transform)

train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
    shuffle=True, num_workers=num_workers)
valid_loader = torch.utils.data.DataLoader(val_data, batch_size=batch_size, 
    shuffle=False, num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,shuffle=False,
    num_workers=num_workers)
</code></pre>

Nikronic: <p>It means the images in your dataset do not have same sizes but I can see you are using <code>transforms.Resize(256)</code>. I really cannot figure out, maybe some of your images are not in same mode such as <code>RGB</code>?</p>
<p>I think it would be much easier if you define your own custom dataset loader. You will have full control over data loading. <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">dataloading tutorial</a></p>
<p>This is a post for similar case: <a href="https://discuss.pytorch.org/t/custom-dataset-with-some-preprocessing/30385/2" class="inline-onebox">Custom Dataset with some preprocessing</a></p>

MonIA: <p>I will check the links you added. Thank you!</p>