ValueError: Expected input batch_size (1) to match target batch_size (64)

username: eadomenech
cooked: <pre><code class="lang-python">from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(64 * 3 * 8 * 8, 1000)
        self.fc2 = nn.Linear(1000, 200)
        self.fc3 = nn.Linear(200, 10)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1)


def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        data = data.view(-1, args.batch_size*3*8*8)
        target = target.view(-1, args.batch_size)
        output = model(data)
        loss = F.nll_loss(output, target[0])
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))


def test(args, model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            data = data.view(-1, args.test_batch_size*3*8*8)
            target = target.view(-1, args.test_batch_size)
            output = model(data)
            # sum up batch loss
            test_loss += F.nll_loss(output, target[0], reduction='sum').item()
            # get the index of the max log-probability
            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print(
        '\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            test_loss, correct, len(test_loader.dataset),
            100. * correct / len(test_loader.dataset)))


def main():
    # Training settings
    parser = argparse.ArgumentParser(description='FNN')
    parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                        help='input batch size for training (default: 64)')
    parser.add_argument(
        '--test-batch-size', type=int, default=64, metavar='N',
        help='input batch size for testing (default: 1000)')
    parser.add_argument('--epochs', type=int, default=10, metavar='N',
                        help='number of epochs to train (default: 10)')
    parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--no-cuda', action='store_true', default=False,
                        help='disables CUDA training')
    parser.add_argument('--seed', type=int, default=1, metavar='S',
                        help='random seed (default: 1)')
    parser.add_argument(
        '--log-interval', type=int, default=1000, metavar='N',
        help='how many batches to wait before logging training status')

    parser.add_argument('--save-model', action='store_true', default=True,
                        help='For Saving the current Model')
    args = parser.parse_args()
    use_cuda = not args.no_cuda and torch.cuda.is_available()

    torch.manual_seed(args.seed)

    device = torch.device("cuda" if use_cuda else "cpu")

    kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
    
    # Transforms
    simple_transform = transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]
    )

    # Dataset
    train_dataset = datasets.ImageFolder('data/train/', simple_transform)
    valid_dataset = datasets.ImageFolder('data/valid/', simple_transform)

    # Data loader
    train_loader = torch.utils.data.DataLoader(
        dataset=train_dataset, batch_size=args.batch_size, shuffle=True,
        num_workers=2)

    test_loader = torch.utils.data.DataLoader(
        dataset=valid_dataset, batch_size=args.test_batch_size, shuffle=False,
        num_workers=2)

    model = Net().to(device)
    optimizer = optim.SGD(
        model.parameters(), lr=args.lr, momentum=args.momentum)

    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        test(args, model, device, test_loader)

    if (args.save_model):
        torch.save(model.state_dict(), "fnn.pt")

if __name__ == '__main__':
    main()
</code></pre>

username: ptrblck
cooked: <p>The <code>view</code> operations of <code>data</code> and <code>target</code> look strange.<br>
Currently you are using</p>
<pre><code class="lang-python">data = data.view(-1, args.test_batch_size*3*8*8)
target = target.view(-1, args.test_batch_size)
</code></pre>
<p>Generally and also based on your model code, you should provide the data as <code>[batch_size, in_features]</code> and the target as <code>[batch_size]</code> containing class indices.<br>
Could you change that and try to run your code again?</p>
<p>PS: I’ve formatted your code for better readability. You can add code snippets using three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: eadomenech
cooked: <p>Thanks for your quick response. Here I mention what exactly I want to do to help as much as possible. I have in <code>data</code> the folders<code>train</code> and <code>valid</code> and within them 9 folders that you would be the classes. They contain RGB images of 8x8 pixels. The idea is to use an MLP to classify future 8x8 images in one of these 9 classes.</p>
<p>I am using batch_size = 8. Here I made some changes according to your comments. You will tell me the necessary changes to make to meet the objective.</p>
<pre><code class="lang-auto">from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(3 * 8 * 8, 1000)
        self.fc2 = nn.Linear(1000, 200)
        self.fc3 = nn.Linear(200, 9)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1)


def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        data = data.view(args.batch_size,  3 * 8 * 8)
        target = target.view(args.batch_size, -1)
        output = model(data)
        loss = F.nll_loss(output, target[0])
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * args.batch_size, len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))


def test(args, model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            data = data.view(args.test_batch_size,  3 * 8 * 8)
            target = target.view(args.test_batch_size, -1)
            output = model(data)
            # sum up batch loss
            test_loss += F.nll_loss(output, target[0], reduction='sum').item()
            # get the index of the max log-probability
            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print(
        '\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            test_loss, correct, len(test_loader.dataset),
            100. * correct / len(test_loader.dataset)))


def main():
    # Training settings
    parser = argparse.ArgumentParser(description='FNN')
    parser.add_argument('--batch-size', type=int, default=8, metavar='N',
                        help='input batch size for training (default: 64)')
    parser.add_argument(
        '--test-batch-size', type=int, default=8, metavar='N',
        help='input batch size for testing (default: 1000)')
    parser.add_argument('--epochs', type=int, default=10, metavar='N',
                        help='number of epochs to train (default: 10)')
    parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--no-cuda', action='store_true', default=False,
                        help='disables CUDA training')
    parser.add_argument('--seed', type=int, default=1, metavar='S',
                        help='random seed (default: 1)')
    parser.add_argument(
        '--log-interval', type=int, default=100, metavar='N',
        help='how many batches to wait before logging training status')

    parser.add_argument('--save-model', action='store_true', default=True,
                        help='For Saving the current Model')
    args = parser.parse_args()
    use_cuda = not args.no_cuda and torch.cuda.is_available()

    torch.manual_seed(args.seed)

    device = torch.device("cuda" if use_cuda else "cpu")

    kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
    
    # Transforms
    simple_transform = transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]
    )

    # Dataset
    train_dataset = datasets.ImageFolder('data/train/', simple_transform)
    valid_dataset = datasets.ImageFolder('data/valid/', simple_transform)

    # Data loader
    train_loader = torch.utils.data.DataLoader(
        dataset=train_dataset, batch_size=args.batch_size, shuffle=True,
        num_workers=2)

    test_loader = torch.utils.data.DataLoader(
        dataset=valid_dataset, batch_size=args.test_batch_size, shuffle=False,
        num_workers=2)

    model = Net().to(device)
    optimizer = optim.SGD(
        model.parameters(), lr=args.lr, momentum=args.momentum)

    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        test(args, model, device, test_loader)

    if (args.save_model):
        torch.save(model.state_dict(), "fnn.pt")

if __name__ == '__main__':
    main()
</code></pre>

username: ptrblck
cooked: <p>I think <code>target</code> should be viewed as <code>target = target.view(args.batch_size)</code>.<br>
Did you try to run the code and checked for other errors?</p>

username: eadomenech
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="43071">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>target = target.view(args.batch_size)</p>
</blockquote>
</aside>
<p>Perfect.<br>
Making <code>target = target.view (args.batch_size)</code> resolves the problem.</p>
<p>Finally the code was as follows:</p>
<pre><code class="lang-auto">from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(3 * 8 * 8, 1000)
        self.fc2 = nn.Linear(1000, 200)
        self.fc3 = nn.Linear(200, 9)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1)


def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        data = data.view(args.batch_size,  3 * 8 * 8)
        target = target.view(args.batch_size)
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * args.batch_size, len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))


def test(args, model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            data = data.view(args.test_batch_size,  3 * 8 * 8)
            target = target.view(args.test_batch_size)
            output = model(data)
            # sum up batch loss
            test_loss += F.nll_loss(output, target, reduction='sum').item()
            # get the index of the max log-probability
            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print(
        '\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            test_loss, correct, len(test_loader.dataset),
            100. * correct / len(test_loader.dataset)))


def main():
    # Training settings
    parser = argparse.ArgumentParser(description='FNN')
    parser.add_argument('--batch-size', type=int, default=8, metavar='N',
                        help='input batch size for training (default: 64)')
    parser.add_argument(
        '--test-batch-size', type=int, default=8, metavar='N',
        help='input batch size for testing (default: 1000)')
    parser.add_argument('--epochs', type=int, default=100, metavar='N',
                        help='number of epochs to train (default: 10)')
    parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--no-cuda', action='store_true', default=False,
                        help='disables CUDA training')
    parser.add_argument('--seed', type=int, default=1, metavar='S',
                        help='random seed (default: 1)')
    parser.add_argument(
        '--log-interval', type=int, default=100, metavar='N',
        help='how many batches to wait before logging training status')

    parser.add_argument('--save-model', action='store_true', default=True,
                        help='For Saving the current Model')
    args = parser.parse_args()
    use_cuda = not args.no_cuda and torch.cuda.is_available()

    torch.manual_seed(args.seed)

    device = torch.device("cuda" if use_cuda else "cpu")

    kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
    
    # Transforms
    simple_transform = transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]
    )

    # Dataset
    train_dataset = datasets.ImageFolder('data/train/', simple_transform)
    valid_dataset = datasets.ImageFolder('data/valid/', simple_transform)

    # Data loader
    train_loader = torch.utils.data.DataLoader(
        dataset=train_dataset, batch_size=args.batch_size, shuffle=True,
        num_workers=2)

    test_loader = torch.utils.data.DataLoader(
        dataset=valid_dataset, batch_size=args.test_batch_size, shuffle=False,
        num_workers=2)

    model = Net().to(device)
    optimizer = optim.SGD(
        model.parameters(), lr=args.lr, momentum=args.momentum)

    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        test(args, model, device, test_loader)

    if (args.save_model):
        torch.save(model.state_dict(), "fnn.pt")

if __name__ == '__main__':
    main()
</code></pre>
<p>Again thanks.</p>

username: mathematics
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Please could you explain in brief why these error occurs ?</p>
<p>Also what are the different case scenario these cases occurs , in simple codes?</p>
<p>Thankyou for answer in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I had got same error but got no idea where could it occur</p>
<details>
<summary>
Summary</summary>
<p>model = Classifier()<br>
criterion = nn.NLLLoss()<br>
optimizer = optim.Adam(model.parameters(), lr=0.003)</p>
<p>epochs = 30<br>
steps = 0</p>
<p>train_losses, test_losses = [], []<br>
for e in range(epochs):<br>
running_loss = 0<br>
for images, labels in trainloader:</p>
<pre><code>    optimizer.zero_grad()
    
    log_ps = model(images)
    loss = criterion(log_ps, labels)
    loss.backward()
    optimizer.step()
    
    running_loss += loss.item()
    
else: #after each loop complete
    test_loss = 0
    train_loss = 0
    
    ## TODO: Implement the validation pass and print out the validation accuracy
    with torch.no_grad():
        for images, labes in testloader:
            log_ps  = model(images)
            test_loss += criterion(log_ps, labels) #update test loss
            
            ps = torch.exp(log_ps) #actual probs
            top_p, top_class = ps.topk(1, dim=1) #gives actual predicted classes
            equals = top_class == labels.view(*top_class.shape) #whether predicted class match with actual class
            accuracy += torch.mean(equals.type(torch.FloatTensor))
    train_losses.append(running_loss/len(trainloader))
    test_losses.append(test_loss/len(testloader))
</code></pre>
</details>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mathematics" data-post="6" data-topic="43071">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathematics/48/40082_2.png" class="avatar"> mathematics:</div>
<blockquote>
<p>Please could you explain in brief why these error occurs ?</p>
<p>Also what are the different case scenario these cases occurs , in simple codes?</p>
</blockquote>
</aside>
<p>This error is raised, if the batch sizes of the model output and target do not match.<br>
E.g. this small code example shows the issue, where the data uses a batch size of 5, while the target uses one of 4:</p>
<pre data-code-wrap="python"><code class="lang-python">
nb_classes = 10
output = torch.randn(5, nb_classes, requires_grad=True) # [batch_size, nb_classes]
target = torch.randint(0, nb_classes, (4,)) # [batch_size]

criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
&gt; ValueError: Expected input batch_size (5) to match target batch_size (4).
</code></pre>
<p>Check the shapes of <code>log_ps</code> and <code>labels</code> in your code and make sure they have the same batch size.</p>

username: mathematics
cooked: <p>oh my stupid fault , I was having typo of labes instead labels and was working previously written labels variable in criterion. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I appretiate. .</p>

username: Ahebwaed
cooked: <p>Hello, am having a similar error ,<br>
Am doing Phase retrieval in digital holography but the error below is messing me up. any help will be greatly appreciated.<br>
Here is CN network, and am using a batch size of 2.</p>
<p>class Network(nn.Module):<br>
def <strong>init</strong>(self, in_channels=2 ):</p>
<pre><code>     super(Network,self).__init__()

     self.Conv1 =nn.Sequential(
            nn.Conv2d(in_channels,16, 3,1,1),nn.ReLU(),
            nn.MaxPool2d(3,1,1),
            nn.Conv2d(16,2, 3),nn.ReLU())
     
     self.Conv2 = nn.Sequential(
            nn.Conv2d(in_channels, 16,3,1,1),nn.ReLU(),
			nn.MaxPool2d(3,1,1),
            nn.Conv2d(16, 4, 3),nn.ReLU())

     self.Conv3 =  nn.Sequential(
	        nn.Conv2d(in_channels, 16,3,1,1),nn.ReLU(),
		    nn.MaxPool2d(3,1,1),
            nn.Conv2d(16, 8, 3),nn.ReLU())
     
     self.Conv4 =nn.Sequential(
	        nn.Conv2d(in_channels, 16, 3,1,1),nn.ReLU(),
			nn.MaxPool2d(3,1,1),
            nn.Conv2d(16,16, 3),nn.ReLU())
           

     self.Upsample1 = nn.Sequential(
              nn.Upsample(size=(256, 256), scale_factor=None,align_corners=True, mode="bilinear"),
              nn.Conv2d(2, 4, 3),nn.ReLU(),
              
              nn.Upsample(size=(256, 256), scale_factor=None,align_corners=True, mode='bilinear'),
              nn.Conv2d(4, 8, 3),nn.ReLU(),
              
              nn.Upsample(size=(256, 256), scale_factor=None,align_corners=True, mode='bilinear'),
              nn.Conv2d(8, 16, 3),nn.ReLU())

     self.Upsample2 = nn.Sequential(
              nn.Upsample(size=(256, 256), scale_factor=None,align_corners=True, mode='bilinear'),
              nn.Conv2d(4, 8, 3),nn.ReLU(),
              
              nn.Upsample(size=(256, 256), scale_factor=None,align_corners=True, mode='bilinear'),
              nn.Conv2d(8, 16, 3),nn.ReLU())              
                
     self.Upsample3 = nn.Sequential(
              nn.Upsample(size=(256, 256), scale_factor=None,align_corners=True, mode='bilinear'),
              nn.Conv2d(8, 16, 3),nn.ReLU())
     
     self.Conv1a = nn.Conv2d(16, 16, 3)
     self.Conv2a = nn.Conv2d(16, 16, 3)                   
     self.Conv3a = nn.Conv2d(16, 16, 3)
     self.Conv4a = nn.Conv2d(16, 16, 3) 
     self.Conv5 =  nn.Conv2d(16, 64, 3)

def forward(self,x):
   # print(x.size)
    w = self.Conv1(x)
    w = self.Upsample1(w)
    w=  F.relu(self.Conv1a(w))

    k = self.Conv2(x)
    k = self.Upsample2(k)
    k=  F.relu(self.Conv2a(k))

    y = self.Conv3(x)
    y = self.Upsample3(y)
    y=  F.relu(self.Conv3a(y))

    z = self.Conv4(x)
    z=  F.relu(self.Conv4a(z))
    print('*****',w.shape,k.shape,y.shape,z.shape)
    
    p = torch.cat([w, k, y, z], 0)
    print(p.shape)
    q = self.Conv5(p)
    print(q.shape)


    return p
</code></pre>
<p>net= Network()</p>
<p>Below is the error am getting…</p>
<p>***** torch.Size([1, 16, 252, 252]) torch.Size([1, 16, 252, 252]) torch.Size([1, 16, 252, 252]) torch.Size([1, 16, 252, 252])<br>
torch.Size([4, 16, 252, 252])<br>
torch.Size([4, 64, 250, 250])<br>
output_shape torch.Size([4, 16, 252, 252])<br>
Traceback (most recent call last):</p>
<p>File “E:\DMU\pytnewCNN.ipy”, line 201, in <br>
loss =criterion(outputs, labels)</p>
<p>File “C:\Users\Hp\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)</p>
<p>File “C:\Users\Hp\anaconda3\lib\site-packages\torch\nn\modules\loss.py”, line 932, in forward<br>
ignore_index=self.ignore_index, reduction=self.reduction)</p>
<p>File “C:\Users\Hp\anaconda3\lib\site-packages\torch\nn\functional.py”, line 2317, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)</p>
<p>File “C:\Users\Hp\anaconda3\lib\site-packages\torch\nn\functional.py”, line 2113, in nll_loss<br>
.format(input.size(0), target.size(0)))</p>
<p><strong>ValueError: Expected input batch_size (4) to match target batch_size (2).</strong></p>

username: ptrblck
cooked: <p>I’m not familiar with your model, but this line of code:</p>
<pre><code class="lang-python">p = torch.cat([w, k, y, z], 0)
</code></pre>
<p>would concatenate the different activations in the batch dimension, which is most likely wrong.<br>
If you would like to concatenate the activations instead in the channel dimension, you would have to use <code>dim=1</code>.<br>
Currently this line is changing the batch size, which is probably creating the error.</p>

username: winterknight
cooked: <p>I’m encountering a similar error.<br>
It trains for almost an entire epoch and fails towards in last few steps.</p>
<p><code>ValueError: Expected input batch_size (1) to match target batch_size (0).</code></p>
<p>I think there is something wrong with the way I’m setting up the dimensions of the linear layers (specifically linear1)  but shouldn’t the model throw an error right at the beginning of training?</p>
<p>The inputs to the <code>MultiheadAttention</code> are enocder and decoder outputs both of dims [2, 1024, 768].<br>
Output of <code>MultiheadAttention</code> is also [2, 1024, 768]<br>
I use <code>self.linear1</code> as <code>self.linear1(multi_attn_output.reshape(-1, 1024*768))</code></p>
<p>The output of <code>self.linear1</code> also looks alright : [2, 512]</p>
<pre><code class="lang-auto">alpha = 0.5
class CrossAttentionSummarizer(pl.LightningModule):
  def __init__(self):
        super(CrossAttentionSummarizer, self).__init__()
        self.summarizer_model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)
        self.qa_encoder = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)
        self.multihead_attn = nn.MultiheadAttention(embed_dim=768, num_heads=4, batch_first=True)
        self.linear1 = nn.Linear(1024*768, 512)
        self.linear2 = nn.Linear(512, 2, bias=False)
        self.ce_loss = nn.CrossEntropyLoss()

  def forward(self, question_passage_input_ids, question_passage_attention_mask, question_labels, input_ids, attention_mask, decoder_attention_mask, labels=None):
    summarizer_output = self.summarizer_model(
        input_ids,
        attention_mask=attention_mask,
        labels=labels,
        decoder_attention_mask=decoder_attention_mask
    )

    qa_output = self.qa_encoder(
        question_passage_input_ids, 
        question_passage_attention_mask,
        question_labels
    )
    decoder_output = summarizer_output[3]
    encoder_output = qa_output[2]

    multi_attn_output, multi_attn_output_weights = self.multihead_attn(decoder_output, encoder_output, encoder_output)
    lin_output = self.linear1(multi_attn_output.reshape(-1, 1024*768))
    cls_outputs = self.linear2(lin_output)
    cls_outputs = nn.functional.softmax(cls_outputs, dim=1)
    cls_preds = torch.argmax(cls_outputs, dim=1)
    cls_pred_loss = self.ce_loss(cls_outputs, question_labels.type(torch.int64).squeeze())
    return summarizer_output.loss, summarizer_output.logits, cls_pred_loss, cls_preds
</code></pre>

username: ptrblck
cooked: <p>Check if you are using <code>squeeze()</code> somewhere without specifying the <code>dim</code> argument, as it seems you are accidentally dropping the batch dimension with a size of 1.</p>

username: winterknight
cooked: <p>I use <code>squeeze()</code> in<br>
<code>cls_pred_loss = self.ce_loss(cls_outputs, question_labels.type(torch.int64).squeeze())</code></p>
<p><code>question_labels</code> are of type int, 0 or 1. And batched by the DataLoader.</p>
<p>Removing that causes the dimension of the classification labels to botch up…<br>
<code>RuntimeError: 0D or 1D target tensor expected, multi-target not supported</code></p>
<p>This is a binary classification. I’m using 2 output heads and applying <code>torch.argmax</code> for preds and <code>CrossEntropyLoss()</code> for the loss</p>

username: ptrblck
cooked: <p>You shouldn’t remove the <code>squeeze</code> operation, but should be explicit in which dimension should be removed.<br>
E.g. if you want to drop <code>dim1</code> use:</p>
<pre><code class="lang-python"># batch_size 2
x = torch.randn(2, 1)

y = x.squeeze(dim=1)
print(y.shape)
# torch.Size([2])


# batch_size 1
x = torch.randn(1, 1)

y = x.squeeze(dim=1)
print(y.shape)
# torch.Size([1])


# batch_size 1
x = torch.randn(1, 1)

y = x.squeeze()
print(y.shape)
# torch.Size([]) !!!!
</code></pre>

username: winterknight
cooked: <p>Trying it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I have one other doubt …<br>
In :<br>
<code>cls_pred_loss = self.ce_loss(cls_outputs, question_labels.type(torch.int64).squeeze(dim=1))</code></p>
<p>the dimension of <code>cls_outputs</code> is [2,2] (batch_first=True) and that of <code>question_labels</code> is [2,1]</p>
<p>So, in <code>CrossEntropyLoss()</code> I’m using the outputs of the 2 logits <code>cls_output</code> and a class label 0/1.<br>
I’m trying to conform to the definition of CELoss() as per the documentation:<br>
<code>input has to be a 2D Tensor of size (minibatch, C).</code><br>
<code>This criterion expects a class index (0 to C-1) as the target for each value of a 1D tensor of size minibatch</code><br>
I hope this implementation is correct (?)</p>
<p>And finally I use <code>dim</code> in  the prediction and loss layer too…</p>
<pre><code class="lang-auto">cls_outputs = nn.functional.softmax(cls_outputs, dim=1)
cls_preds = torch.argmax(cls_outputs, dim=1)
</code></pre>