RuntimeError: expected type torch.FloatTensor but got torch.cuda.FloatTensor

rahulthen: <p>After loading the model from check point . I tried to train the model again. While training it shows error on optimizer.step()</p>
<pre><code class="lang-auto">def load_checkpoint(checkpoint_path):
    
    model = models.vgg11(pretrained=True)
    for param in model.parameters():
        param.requires_grad = False
    
    checkpoint = torch.load(path)
    
    classifier = nn.Sequential(OrderedDict([('fc1', nn.Linear(checkpoint['input_size'], 6272)),
                                        ('relu', nn.ReLU()),
                                        ('drop1',nn.Dropout(p=0.4)),
                                        ('fc2', nn.Linear(6272, 512)),
                                        ('relu', nn.ReLU()),
                                        ('drop2',nn.Dropout(p=0.2)),
                                        ('fc3', nn.Linear(512, checkpoint['out_size'])),
                                        ('output', nn.LogSoftmax(dim=1))]))
    
    model.classifier = classifier
    model.classifier.load_state_dict(checkpoint['state_dict'])
    #model.classifier.fc1.in_features = checkpoint['input_size']
    #model.classifier.fc3.out_features = checkpoint['out_size']
    class_to_idx = checkpoint['class_to_idx']
    
    optimizer = optim.Adam(model.classifier.parameters(), lr = 0.001)
    optimizer.load_state_dict(checkpoint['optim_state'])
    
    return model, optimizer, class_to_idx

model, optimizer, class_to_idx = load_checkpoint(path)

idx_to_class = { v : k for k,v in class_to_idx.items()}

</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d84319ccd0de0177ae9a1cad6bad8a68cb0c687b.png" data-download-href="https://discuss.pytorch.org/uploads/default/d84319ccd0de0177ae9a1cad6bad8a68cb0c687b" title="Screenshot%20from%202019-02-16%2021-41-37"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d84319ccd0de0177ae9a1cad6bad8a68cb0c687b_2_690x388.png" alt="Screenshot%20from%202019-02-16%2021-41-37" data-base62-sha1="uR8TzG3bmq2ykvbJhZ3wf7KCXJp" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d84319ccd0de0177ae9a1cad6bad8a68cb0c687b_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d84319ccd0de0177ae9a1cad6bad8a68cb0c687b_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d84319ccd0de0177ae9a1cad6bad8a68cb0c687b_2_1380x776.png 2x" data-dominant-color="F0EFF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-02-16%2021-41-37</span><span class="informations">1920×1080 216 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-7-5e78ed847bd0&gt; in &lt;module&gt;()
     16         loss = criterion(logps, labels)
     17         loss.backward()
---&gt; 18         optimizer.step()
     19 
     20         train_loss += loss.item()

/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py in step(self, closure)
     91 
     92                 # Decay the first and second moment running average coefficient
---&gt; 93                 exp_avg.mul_(beta1).add_(1 - beta1, grad)
     94                 exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
     95                 if amsgrad:

RuntimeError: expected type torch.FloatTensor but got torch.cuda.FloatTensor```</code></pre>

InnovArul: <p>Make sure all the relevant variables are in GPU.</p>
<p>Can you try to push the <code>optimizer</code> to gpu as well,  before training loop?</p>
<pre><code class="lang-auto">optimizer = optimizer.cuda()
</code></pre>

sab: <p><a class="mention" href="/u/rahulthen">@rahulthen</a> how did you fix this?</p>

Bananin: <p>I had a similar issue and I think it is related to your optimizer referencing model parameters that are not yet loaded on GPU. In my case I fixed the issue by changing the following:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=params.lr)
try:
       checkpoint = torch.load("checkpoints/"+model_folder+"checkpoint.pth.tar")
       start_epoch = checkpoint['epoch']
       scheduler.load_state_dict(checkpoint['scheduler'])
       model.load_state_dict(checkpoint['state_dict'])
       model.to(device) # moving this line here fixed the issue
       optimizer.load_state_dict(checkpoint['optimizer'])
       …
       # it used to be somewhere around here
</code></pre>
<p>So it seems you need to send de <strong>model</strong> to GPU BEFORE loading the optimizer dictionary, else it will be loaded to the usual memory instead of GPU memory, which caused your issue (I think).</p>

fateme_azeami: <p>thanks! that does the job!</p>