Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

Daanwur: <p>Hi everyone,<br>
I’m training a model using PyTorch and while running the train function I encounter the following error message:</p>
<blockquote>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</p>
</blockquote>
<p>During the run, I noticed that after the first epoch my tensor changes its working device to CPU from the GPU, as can be seen here:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb333ee5dc25d18dd0dd4b360bf73fb44e565ecf.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb333ee5dc25d18dd0dd4b360bf73fb44e565ecf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb333ee5dc25d18dd0dd4b360bf73fb44e565ecf_2_690x309.png" alt="image" data-base62-sha1="qI34YW9T7ORUpkcK1JUQLh2zKXB" width="690" height="309" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb333ee5dc25d18dd0dd4b360bf73fb44e565ecf_2_690x309.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb333ee5dc25d18dd0dd4b360bf73fb44e565ecf_2_1035x463.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb333ee5dc25d18dd0dd4b360bf73fb44e565ecf_2_1380x618.png 2x" data-dominant-color="312E30"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1917×860 146 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would really appreciate your help,<br>
thank you in advance</p>

ptrblck: <p>Check if the data was properly moved to the GPU as this error indicates a device mismatch in the model execution while the parameters of the model seem to be on the GPU already.</p>

Daanwur: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="152430">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>moved to t</p>
</blockquote>
</aside>
<p>At each training iteration I’m moving both the data and the model to the GPU as in the attached code:</p>
<blockquote>
<p>def train(num_epochs, model, optimizer, loss_fn, train_loader):<br>
best_accuracy = 0.0<br>
# Define your execution device<br>
device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
print(“The model will be running on”, device, “device”)<br>
# Convert model parameters and buffers to CPU or Cuda</p>
</blockquote>
<pre><code>for epoch in range(num_epochs):  # loop over the dataset multiple times
    running_loss = 0.0
    running_acc = 0.0

    for i, (images, labels) in enumerate(tqdm(train_loader, 0)):

        model = model.to(torch.device('cuda'))
        # get the inputs
        images = images.to(torch.device('cuda'))
        labels = labels.to(torch.device('cuda'))

        # zero the parameter gradients
        optimizer.zero_grad()
        # predict classes using images from the training set
        outputs = model(images)
        # compute the loss based on model output and real labels
        loss = loss_fn(outputs, torch.max(labels, 1)[1])
        # backpropagate the loss
        loss.backward()
        # adjust parameters based on the calculated gradients
        optimizer.step()

        # Let's print statistics for every 1,000 images
        running_loss += loss.item()  # extract the loss value
        if i % 10 == 0:
            # print every 1000 (twice per epoch)
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 1000))
            # zero the loss
            running_loss = 0.0
        sleep(0.1)
    # Compute and print the average accuracy fo this epoch when tested over all 10000 test images
    accuracy = test_accuracy(model, train_loader)
    print('For epoch', epoch + 1, 'the test accuracy over the whole test set is %d %%' % (accuracy))

    # we want to save the model if the accuracy is the best
    if accuracy &gt; best_accuracy:
        save_model()
        best_accuracy = accuracy
</code></pre>
<p>Or should I do it before? while creating the Dataset ?</p>

ptrblck: <p>Your code looks correct and you could remove the:</p>
<pre><code class="lang-python">model = model.to(torch.device('cuda'))
</code></pre>
<p>from the <code>DataLoader</code> loop as the model should be moved once to the device before the training starts.</p>
<p>Are you creating any tensors in the <code>forward</code> method without moving them to the GPU and could you also check the validation or test loop and make sure the data is also moved to the GPU there?<br>
If you get stuck, could you post a minimal, executable code snippet reproducing the issue, please?</p>

Daanwur: <p>If I understand you correctly as I create the Dataloader should I immediately move it to the GPU?</p>
<blockquote>
<p>train_set = Dataset.CtDataset(x_train, y_train, transform=train_transform, kind=‘train’)<br>
val_set = Dataset.CtDataset(x_test, y_test, transform=val_transform, kind=‘val’)</p>
<p>print(‘Train size: {}’.format(len(train_set)))<br>
print(‘Test size: {}’.format(len(val_set)))</p>
<p>train_loader = DataLoader(train_set, batch_size=64, shuffle=True)<br>
valid_loader = DataLoader(val_set, batch_size=64, shuffle=True)</p>
</blockquote>
<p>I will try to see if it works during test or validation, the line in which I get the error is in bold and occurs right after the first epoch is about to end. I tried moving my input within the forward method to the GPU but there was no effect… I’m attaching my forward method with the whole model class:</p>
<blockquote>
<p>class Model(nn.ModuleList):</p>
<pre><code>def __init__(self):
    super(Model, self).__init__()
    self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=5, stride=1, padding=1)
    self.bn1 = nn.BatchNorm2d(12)
    self.conv2 = nn.Conv2d(in_channels=12, out_channels=12, kernel_size=5, stride=1, padding=1)
    self.bn2 = nn.BatchNorm2d(12)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv4 = nn.Conv2d(in_channels=12, out_channels=24, kernel_size=5, stride=1, padding=1)
    self.bn4 = nn.BatchNorm2d(24)
    self.conv5 = nn.Conv2d(in_channels=24, out_channels=24, kernel_size=5, stride=1, padding=1)
    self.bn5 = nn.BatchNorm2d(24)
    self.fc1 = nn.Linear(80736, 64)

def forward(self, x):
    print(x.device)
    **x = F.relu(self.bn1(self.conv1(x)))**
    x = F.relu(self.bn2(self.conv2(x)))
    x = self.pool(x)
    x = F.relu(self.bn4(self.conv4(x)))
    x = F.relu(self.bn5(self.conv5(x)))
    x = x.view(x.size(0), -1)
    x = self.fc1(x)
    return x
</code></pre>
</blockquote>
<p>Thank you so much for helping me!</p>

ptrblck: <p>Based on the location of the error the input doesn’t seem to be moved to the device and the <code>forward</code> method looks aright.<br>
Check if:</p>
<pre><code class="lang-python">images = images.to(torch.device('cuda'))
labels = labels.to(torch.device('cuda'))
</code></pre>
<p>is used in all <code>DataLoader</code> loops (training, validation, test, etc.).</p>

Daanwur: <p>I think I managed to find the problem, while sending the Dataloader to the accuracy function I did not change the device which may cause the crash. checking it now, will update shortly.<br>
thank you !</p>

Daanwur: <p>I managed to find what the problem is - as I mentioned in another comment of mine the problem came from the Dataloader - I didn’t move it onto the GPU in the accuracy function. After doing so I’m now getting the following Error:</p>
<blockquote>
<p>RuntimeError: The size of tensor a (64) must match the size of tensor b (7) at non-singleton dimension 1</p>
</blockquote>
<p>The function:</p>
<blockquote>
<p>def test_accuracy(model, test_loader):<br>
model.eval()<br>
acc = 0.0<br>
total = 0.0</p>
<pre><code>with torch.no_grad():
    for data in test_loader:
        images, labels = data
        images = images.to(torch.device('cuda'))
        labels = labels.to(torch.device('cuda'))
        # run the model on the test set to predict labels
        outputs = model(images)
        # the label with the highest energy will be our prediction
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        acc += (predicted == labels).sum().item()

# compute the accuracy over all test images
acc = (100 * acc / total)
return (acc)
</code></pre>
</blockquote>

ptrblck: <p>I guess the error is raised in the accuracy calculation:</p>
<pre><code class="lang-python">predicted = torch.randint(0, 10, (2, 64))
labels = torch.randint(0, 10, (2, 7))
(predicted == labels).sum().item()
# RuntimeError: The size of tensor a (64) must match the size of tensor b (7) at non-singleton dimension 1
</code></pre>
<p>so check the shapes of these tensors and make sure you can compare them.</p>

Nada_Elmaghraby: <p>I have the same error and I don’t know what to do , this is my code:<br>
<span class="hashtag">#prediction</span> function<br>
def prediction(img_path,transformer):</p>
<pre><code>image=Image.open(img_path)

image_tensor=transformer(image).float()



image_tensor=image_tensor.unsqueeze_(0)

if torch.cuda.is_available():
    image_tensor.cuda()
    
input=Variable(image_tensor)


output=model(input)

index=output.data.numpy().argmax()

pred=classes[index]


return pred
</code></pre>
<p>images_path=glob.glob(pred_path+‘/*.jpg’)<br>
pred_dict={}<br>
for i in images_path:<br>
pred_dict[i[i.rfind(‘/’)+1:]]=prediction(i,transformer)</p>

ptrblck: <p>This code:</p>
<pre><code class="lang-python">if torch.cuda.is_available():
    image_tensor.cuda()
</code></pre>
<p>won’t work as you need to reassign the <code>image_tensor</code> via:</p>
<pre><code class="lang-python">if torch.cuda.is_available():
    image_tensor = image_tensor.cuda()
</code></pre>

praveer_kumar: <p>I also  encounter the same error :<br>
<strong>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</strong></p>
<p>My code is here :</p>
<pre><code class="lang-auto">history += fit(EPOCHS, LR, model, train_dl, val_dl, torch.optim.Adam)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase 
        model.train()
        train_losses = []
        for batch in train_loader:
            loss = model.training_step(batch)
            train_losses.append(loss)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        result['train_loss'] = torch.stack(train_losses).mean().item()
        model.epoch_end(epoch, result)
        history.append(result)
    return history
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  could you please help me on this ?<br>
It fails here :<br>
result = evaluate(model, val_loader)<br>
looks like there is something wrong with val_loader</p>

praveer_kumar: <p>I managed to resolve this issue. I had missed to load val_data to the GPU. and hence it was throwing the error</p>

pdc_87: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Could you help me with these issue?</p>
<p>I’ve moved data, model and loss func to cuda, but it seems the input tensor is not switching to cuda</p>
<p>The model is a CNN with a embedded quantum circuit.</p>
<p><strong>RuntimeError</strong> : Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</p>
<pre><code class="lang-auto">train_data = torchvision.datasets.ImageFolder(path_to_train', transform=transforms.Compose([transforms.ToTensor()]))
test_data = torchvision.datasets.ImageFolder('path_to_test', transform=transforms.Compose([transforms.ToTensor()]))

train_loader = DataLoader(train_data, shuffle=True, batch_size=1)
test_loader = DataLoader(test_data, shuffle=True, batch_size=1)

for data, target in train_loader:
    data = data.to(device)
    target = target.to(device)

for data, target in test_loader:
    data = data.to(device)
    target = target.to(device)

class Net(Module):
    def __init__(self, qnn):
        super().__init__()
        self.conv1 = Conv2d(3, 1, kernel_size=5)
        self.conv2 = Conv2d(1, 1, kernel_size=5)
        self.dropout = Dropout2d()
        self.fc1 = Linear(3844, 64)
        self.fc2 = Linear(64, 2)  # 2-dimensional input to QNN
        self.qnn = TorchConnector(qnn) # Apply torch connector, weights chosen
        # uniformly at random from interval [-1,1].
        self.fc3 = Linear(1, 1)  # 1-dimensional output from QNN

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2)
        x = self.dropout(x)
        x = x.view(x.shape[0], -1))
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.qnn(x)  # apply QNN
        x = self.fc3(x)
        return cat((x, 1 - x), -1)


model = Net(qnn)

model = model.to('cuda')

# Define model, optimizer, and loss function
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_func = NLLLoss().to('cuda')

# Start training
epochs = 10  # Set number of epochs
loss_list = []  # Store loss history
model.train()  # Set model to training mode

for epoch in range(epochs):
    total_loss = []
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad(set_to_none=True)  # Initialize gradient
        print(data.type, target.type)
        output = model(data)  # Forward pass
        loss = loss_func(output, target)  # Calculate loss
        loss.backward()  # Backward pass
        optimizer.step()  # Optimize weights
        total_loss.append(loss.item())  # Store loss
    loss_list.append(sum(total_loss) / len(total_loss))
    print("Training [{:.0f}%]\tLoss: {:.4f}".format(100.0 * (epoch + 1) / epochs, loss_list[-1]))

</code></pre>

pdc_87: <p>Sorry,lacks this</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
</code></pre>

ptrblck: <aside class="quote no-group" data-username="pdc_87" data-post="14" data-topic="152430">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pdc_87/48/54318_2.png" class="avatar"> pdc_87:</div>
<blockquote>
<p>I’ve moved data, model and loss func to cuda, but it seems the input tensor is not switching to cuda</p>
</blockquote>
</aside>
<p>You’ve moved the temp. data in a dummy loop to the GPU:</p>
<pre data-code-wrap="python"><code class="lang-python">for data, target in train_loader:
    data = data.to(device)
    target = target.to(device)
</code></pre>
<p>but not the actually used data in the training loop:</p>
<pre data-code-wrap="python"><code class="lang-python">for epoch in range(epochs):
    total_loss = []
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad(set_to_none=True)  # Initialize gradient
        print(data.type, target.type)
        output = model(data)  # Forward pass
        ...
</code></pre>
<p>Add <code>data = data.to(device)</code> inside the training loop and it should work.</p>

pdc_87: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Yes! I figured it out! Thanks anyway!!!<br>
the input tensor has shape [3, 260, 260] standing for the RGB channels, height and width of the images.</p>
<p>Do you consider that I can change something into the CNN to improve the accuracy? Any rapid insight that you can share?</p>
<p>Looking to hearing from you.</p>
<p>Best,</p>

ptrblck: <p>I’m not familiar with your use case so cannot comment on the model architecture. However, since you are using <code>nn.NLLLoss</code> make sure to pass log probabilities to this loss function via <code>F.log_softmax</code>.</p>

pd3rvr: <p>Ive seen this problem on porting code from from CUDA to MPS and solved it as below;<br>
For the sake of  other coders who may be less familiar with various platforms, I think a generic answer for this would be, aside adding images and labels to device, also add your model in same way. Example below;</p>
<p>images=images.to(devc)<br>
labels=labels.to(devc)<br>
model = model.to(devc)</p>
<p>before doing;<br>
outputs = model(images)</p>

Abdullah123: <p>I was getting the same mistake even though I was assigned a device to images, labels, and outputs, later I found that I doing<code>--&gt; images.to(device)   instead of doing --&gt; images = images.to(device) </code><br>
in short, don’t forget to re-assigned data and model to the original variable</p>