Can't use BatchNorm1d despite using a batch size of 64 for training

username: Adwaitt
cooked: <p>Hello guys, you can find my code below. It throws the following error “Expected more than 1 value per channel when training, got input size torch.Size([1, 256])”</p>
<p>This is my data generator code:</p>
<pre><code class="lang-auto">class datagen(Dataset):
    def __init__(self, df, tfms):
        super(datagen, self).__init__()
        self.x = df['x']
        self.y = df['y']
        self.tfms = tfms
    def __len__(self):
        return len(self.x)
    def __getitem__(self, i):
        img = imageio.imread(self.x[i])
        stacked_img = np.repeat(img[..., np.newaxis], 3, -1)
        x = self.tfms(image = stacked_img)
        y = self.y[i]
        return (x, y)

train_set = datagen(train_df, ttfms)
train = iter(DataLoader(train_set, batch_size = 64, shuffle = False, drop_last = True, num_workers = 16, pin_memory = True))

val_set = datagen(val_df, vtfms)
val = iter(DataLoader(val_set, batch_size = 64, shuffle = False, drop_last = True, num_workers = 16, pin_memory = True))
</code></pre>
<p>This is my code for network and training:</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv = models.resnet101(pretrained = True)
        
        for param in self.conv.parameters():
            param.requires_grad = False
            
        self.fc1 = nn.Sequential(nn.Linear(64000, 2048),
                                 nn.Linear(2048, 1024),
                                 nn.Linear(1024, 256))
        
        self.bn1 = nn.BatchNorm1d(256)
        
        self.fc2 = nn.Sequential(nn.Linear(256, 64),
                                 nn.Linear(64, 16))
        
        self.bn2 = nn.BatchNorm1d(16)
        
        self.out = nn.Linear(16, 1)

    def forward(self, x):
        x = self.conv(x)
        x = x.view(1, 64000)
        x = F.relu(self.bn1(self.fc1(x)))
        x = self.bn2(self.fc2(x))
        x = F.sigmoid(self.out(x))
        return x

model = Network()
model = model.to(device)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr = 0.07)

for e in tqdm(range(10)):
    x, y = train.next()
    x, y = x['image'].float().to(device), y.float().to(device)
    optimizer.zero_grad()
    ps = model.forward(x)
    train_loss = criterion(ps, y.reshape(-1, 1))
    train_loss.backward()
    optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>This <code>view</code> operations looks wrong:</p>
<pre><code class="lang-python">x = x.view(1, 64000)
</code></pre>
<p>since you are flattening the tensor such that it’ll have a batch size of <code>1</code>.<br>
To keep the batch size equal, use <code>x = x.view(x.size(0), -1)</code> instead.</p>
<p>In case you would still run into the batchnorm issue, check if the last batch returned by the <code>DataLoader</code> might be smaller. If that’s the case you can drop it via <code>drop_last=True</code>.</p>

username: Adwaitt
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="131349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>x.size(0), -1</p>
</blockquote>
</aside>
<p>Thank you, you have always helped me and everyone <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>