Trouble with data types in training

motviybletb: <p>Hi everyone!<br>
I have a problem with training a model.<br>
Here is my train function:</p>
<pre><code class="lang-auto">def train(model, optimizer, scheduler, loss_fn, score_fn, epochs, data_tr, data_vl, device):

    torch.cuda.empty_cache()

    losses_train = []
    losses_val = []
    scores_train = []
    scores_val = []

    for epoch in range(epochs):
        tic = time()
        print('* Epoch %d/%d' % (epoch+1, epochs))

        avg_loss = 0
        model.train()  # train mode
        for X_batch, Y_batch in data_tr:
            # data to device
            X_batch = X_batch.to(device)
            Y_batch = Y_batch.to(device)            

            # set parameter gradients to zero
            optimizer.zero_grad()

            # forward
            print(type(X_batch))
            Y_pred = model(X_batch)
            #print("Y_pred len: ", Y_pred.size())
            #print("Y_batch len: ", Y_batch.size())
            loss = loss_fn(Y_pred, Y_batch) # forward-pass

            loss.backward()  # backward-pass
            optimizer.step()  # update weights

            # calculate loss to show the user
            avg_loss += loss / len(data_tr)

        toc = time()
        print('train_loss: %f' % avg_loss)
        losses_train.append(avg_loss)

        # train score
        avg_score_train = score_fn(model, iou_pytorch, data_tr)
        scores_train.append(avg_score_train)

        # val loss
        avg_loss_val = 0
        model.eval()  # testing mode
        for X_val, Y_val in data_vl:
            with torch.no_grad():
                Y_hat = model(X_val.to(device)).detach().cpu()# detach and put into cpu

                loss = loss_fn(Y_hat, Y_val) # forward-pass
                avg_loss_val += loss / len(data_vl)

        toc = time()
        print('val_loss: %f' % avg_loss_val)
        losses_val.append(avg_loss_val)

        # val score
        avg_score_val = score_fn(model, iou_pytorch, data_vl)
        scores_val.append(avg_score_val)

        if scheduler:
            #scheduler.step(avg_score_val)
            scheduler.step()

        torch.cuda.empty_cache()
          
    return (losses_train, losses_val, scores_train, scores_val)
</code></pre>
<p>Here in this function I move data to GPU<br>
After I move my model also to GPU</p>
<pre><code class="lang-auto">unet2_model = UNet2().to(DEVICE)
max_epochs = 80
optimizer = torch.optim.AdamW(unet2_model.parameters(), lr=0.00100, weight_decay=0.07)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[10,20,25,30,40,60,65,70], gamma=0.8)
results_unet_new = train(unet2_model, optimizer, scheduler, bce_loss, score_model, max_epochs, data_tr, data_val, DEVICE)
</code></pre>
<p>So input and weights have to be on one device (GPU), but I see an error “RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor”</p>
<p>If I don’t move model to GPU, I see an error “Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same”</p>
<pre><code class="lang-auto">unet2_model = UNet2()
</code></pre>
<p>Can someone explain me why there is a difference in input type when I just change device of model?</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue as I don’t see any obvious issue in your posted code but also some methods are undefined?</p>

motviybletb: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/fff6849765796459be9010374e46991fcd3e049d.png" data-download-href="https://discuss.pytorch.org/uploads/default/fff6849765796459be9010374e46991fcd3e049d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff6849765796459be9010374e46991fcd3e049d_2_690x388.png" alt="image" data-base62-sha1="AwlURJybG14Q0MJAUw8oVaIo2LP" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff6849765796459be9010374e46991fcd3e049d_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff6849765796459be9010374e46991fcd3e049d_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff6849765796459be9010374e46991fcd3e049d_2_1380x776.png 2x" data-dominant-color="25272A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 250 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here is when I start train<br>
So here you can see error appears after model.eval()<br>
Other parts of code are uploading data, defining functions of losses and model score<br>
So I don’t know why I see this problem</p>

motviybletb: <p>And as I mentioned before: if I change unet2_model = UNet2().to(DEVICE) to unet2_model = UNet2(), I’ll see error “Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same” as soon as I launched my code<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee6b8779d0769922a4d8d365172d82491619736b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee6b8779d0769922a4d8d365172d82491619736b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee6b8779d0769922a4d8d365172d82491619736b_2_690x388.png" alt="image" data-base62-sha1="y1a182vfJi0AHSf3CSLs5ko9gdd" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee6b8779d0769922a4d8d365172d82491619736b_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee6b8779d0769922a4d8d365172d82491619736b_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee6b8779d0769922a4d8d365172d82491619736b_2_1380x776.png 2x" data-dominant-color="25272A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 248 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Posting screenshots isn’t helpful, so please post a minimal and executable code snippet we could use to reproduce and debug the issue.</p>