[Resolved] RuntimeError: expected device cpu and dtype Float but got device cuda:0 and dtype Float

111137: <p>Hi,</p>
<p>My code wok fine on Colabolatory with CUDA.<br>
Now I try to do on local environment.<br>
My env is;</p>
<pre><code class="lang-auto">CUDA: 10.1
cuDNN: 7
</code></pre>
<p>and today I installed PyTorch follows instruction of pip3.<br>
After the setting up I meet an error;</p>
<pre><code class="lang-python">RuntimeError: expected device cpu and dtype Float but got device cuda:0 and dtype Float
</code></pre>
<p>Acutually my code set device(“cuda”), I want it run on NVIDIA GPU.<br>
I can not understand reason why script try to run on CPU, while Colab has no error (indeed run on cpu).<br>
In addition, I replaced operators with torch’s operators, but still I get the error.<br>
Could you tell me reason and its solution?</p>

spanev: <p>Hi,<br>
Can you share the concerned part of the code with us?</p>

111137: <p>Hi,</p>
<p>Thank you for your replying.<br>
Error message is;</p>
<pre><code class="lang-python">Traceback (most recent call last):
  File "QNet.py", line 203, in &lt;module&gt;
    out = model()
  File "/home/syouyu/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "QNet.py", line 164, in forward
    fw_prop(self)
  File "QNet.py", line 70, in fw_prop
    q_layer(self)
  File "QNet.py", line 60, in q_layer
    q_sel(self)
  File "QNet.py", line 53, in q_sel
    self.sel[index] = torch.sigmoid(torch.add(torch.mul(self.w_x[index], self.x[index]), torch.mul(self.w_h[index], self.h[index])))
RuntimeError: expected device cpu and dtype Float but got device cuda:0 and dtype Float
</code></pre>
<p>Tensors are defined as;</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        self.w_x   = [torch.randn((NUM_INPUT), requires_grad=True) for _ in range(NUM_HIDDEN)]
        self.w_h   = [torch.randn((NUM_INPUT), requires_grad=True) for _ in range(NUM_HIDDEN)]
        self.sel   = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]
        self.x     = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]
        self.h     = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]
</code></pre>

spanev: <p>When you are converting your model with <code>model.cuda()</code>, the <code>Tensor</code>'s you have will not be converted: only children <strong>Modules</strong> will be automatically converted.</p>
<p>You will have to manually specify the device of your tensors in the creation:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, device):
        super().__init__()
        self.w_x   = [torch.randn((NUM_INPUT), requires_grad=True, device=device) for _ in range(NUM_HIDDEN)]
        self.w_h   = [torch.randn((NUM_INPUT), requires_grad=True, device=device) for _ in range(NUM_HIDDEN)]
        self.sel   = [torch.zeros(NUM_INPUT, device=device) for _ in range(NUM_HIDDEN)]
        self.x     = [torch.zeros(NUM_INPUT, device=device) for _ in range(NUM_HIDDEN)]
        self.h     = [torch.zeros(NUM_INPUT, device=device) for _ in range(NUM_HIDDEN)]
</code></pre>
<p>so you pass it to your model as:</p>
<pre><code class="lang-auto">model = Model(device='cuda:0')
</code></pre>

111137: <p><a class="mention" href="/u/spanev">@spanev</a> -san,</p>
<p>I do as follows;</p>
<pre><code class="lang-python">device = device_("cuda" if cuda.is_available() else "cpu")
model = Model().to(device)
</code></pre>
<p>And the Model does not have argument of the <em>device</em>.</p>

spanev: <aside class="quote no-group quote-modified" data-username="111137" data-post="5" data-topic="54783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111137/48/15236_2.png" class="avatar"> 111137:</div>
<blockquote>
<p>device = device_(“cuda” if cuda.is_available() else “cpu”)</p>
</blockquote>
</aside>
<p>What is <code>device_</code> here?</p>
<p>You cannot change the model <code>__init__</code> and add a <code>device</code> argument? As done here:</p>
<aside class="quote no-group quote-modified" data-username="spanev" data-post="4" data-topic="54783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spanev/48/14892_2.png" class="avatar"> spanev:</div>
<blockquote>
<p>def <strong>init</strong>(self, device):</p>
</blockquote>
</aside>
<p>?</p>
<p>Also about the Colab you are running on, the default Tensor device might have been set with:</p>
<pre><code class="lang-auto">torch.set_default_tensor_type(torch.cuda.FloatTensor)
</code></pre>
<p>but I wouldn’t recommend relying on this.</p>

111137: <aside class="quote no-group" data-username="spanev" data-post="6" data-topic="54783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spanev/48/14892_2.png" class="avatar"> spanev:</div>
<blockquote>
<p>What is <code>device_</code> here?</p>
</blockquote>
</aside>
<p>It is;</p>
<pre data-code-wrap="python"><code class="lang-python">from torch import device as device_
</code></pre>
<pre><code class="lang-auto">And the Model does not have argument of the device.
</code></pre>
<p>I mean is when having the device as an argument, how using it in the init.</p>

ptrblck: <p>Since you are initializing the tensors with <code>requires_grad=True</code>, you should wrap them into <code>nn.Parameter</code>, so that they will be properly registered in the <code>state_dict</code> and will be automatically pushed to the device, if you call <code>model.to(device)</code>.</p>
<p>Also, since you are storing these parameters in a list, use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.ParameterList" rel="nofollow noopener"><code>ParameterList</code></a>, as a plain Python list won’t register the parameters properly.</p>
<p>This should work:</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.w_x   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.w_h   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.sel   = nn.ParameterList([nn.Parameter(torch.zeros(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.x     = nn.ParameterList([nn.Parameter(torch.zeros(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.h     = nn.ParameterList([nn.Parameter(torch.zeros(NUM_INPUT)) for _ in range(NUM_HIDDEN)])

model = Model()
model.cuda()
print(model.parameters())
</code></pre>

111137: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> -san,</p>
<p>I updated code with your advices, and I got other error;</p>
<pre><code class="lang-python">  File "QNet.py", line 62, in q_sel
    self.sel[index] = torch.sigmoid(torch.add(torch.mul(self.w_x[index], self.x[index]), torch.mul(self.w_h[index], self.h[index])))
RuntimeError: expected device cuda:0 and dtype Float but got device cpu and dtype Float
</code></pre>
<p>Does this mean that my setting on local PC is incorrect ?</p>

ptrblck: <p>I’m not sure, what went wrong, but try to check the device of all parameters after calling <code>model.to('cuda')</code>:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    if param.device.type != 'cuda':
        print('param {}, not on GPU'.format(name))
</code></pre>
<p>This should print, if a parameter is not on the GPU.</p>

111137: <aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="54783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This should print, if a parameter is not on the GPU.</p>
</blockquote>
</aside>
<p>Didn’t print parameters, so  those are probably on GPU card.</p>
<p>Ah, simply</p>
<pre data-code-wrap="python"><code class="lang-python">print(model.parameters())
</code></pre>
<p>also not worked. So, before the run the model, elaboration of function “q_sel” made the error, indeed.</p>

ptrblck: <aside class="quote no-group" data-username="111137" data-post="11" data-topic="54783">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111137/48/15236_2.png" class="avatar"> 111137:</div>
<blockquote>
<p>So, before the run the model, elaboration of function “q_sel” made the error, indeed.</p>
</blockquote>
</aside>
<p>I’m not sure, if I understand the sentence correctly, but is it working now?</p>

111137: <p>Not work.</p>
<p>I use jupyter notebook. Running cell of class Model(), it calls the “q_sel” function of;</p>
<pre><code class="lang-python">File "QNet.py", line 63, in q_sel
    self.sel[index] = torch.sigmoid(self.w_x[index] * self.x[index] + self.w_h[index] * self.h[index])
</code></pre>
<p>Then makes the error.</p>

ptrblck: <p>If you initialize the model, this function is called?<br>
Could you post a reproducible code snippet so that we can have a look?</p>

111137: <p>Not reproducible;</p>
<pre><code class="lang-python">#!/usr/bin/env python
# coding: utf-8

from torchvision import datasets, transforms, models
import torch 
from torch import nn, optim, utils, device as device_, cuda
import numpy as np

# Architecture Hyper-Parameters
NUM_INPUT      = 28
TIME_STEPS     = 28
NUM_CLASS      = 10
NUM_HIDDEN     = 28

def q_sel (self):
  for index in range(NUM_HIDDEN):
    self.sel[index] = torch.sigmoid(self.w_x[index] * self.x[index] + self.w_h[index] * self.h[index])

def q_layer (self):
  q_sel(self)

def fw_prop (self):
  q_layer(self)

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        
        # Gate-Weight
        self.w_x   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.w_h   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
    
        # Gate-Selector
        self.sel   = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]
        
        # Input Vector
        self.x     = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]

        # Output Vector
        self.h     = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]

    def forward(self):
      fw_prop(self)

model = Model()
model.cuda()

print(model.parameters())
for name, param in model.named_parameters():
    if param.device.type != 'cuda':
        print('param {}, not on GPU'.format(name))
</code></pre>
<p>output is</p>
<pre><code class="lang-python">&lt;generator object Module.parameters at 0x7fedc4fd9b48&gt;
</code></pre>
<p>So properly works.</p>
<p>I identified where made the error with print() on original code.<br>
Calling model() after model.train(), it also called function of</p>
<pre><code class="lang-python">def q_sel (self):
  for index in range(NUM_HIDDEN):
    print("2")
    self.sel[index] = torch.sigmoid(self.x[index] * self.w_x[index] + self.h[index] * self.w_h[index])
    print("3")
</code></pre>
<p>After first <code>print("2")</code> getting error. So I think the code could not send data of self.x, self.w_x, self.h, and or self.w_h.</p>

111137: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> -san<br>
I did it;</p>
<pre><code class="lang-python">#!/usr/bin/env python
# coding: utf-8

from torchvision import datasets, transforms, models
import torch 
from torch import nn, optim, utils, device as device_, cuda
import numpy as np

# Architecture Hyper-Parameters
NUM_INPUT      = 28
TIME_STEPS     = 28
NUM_CLASS      = 10
NUM_HIDDEN     = 28

BATCH_SIZE     = 1024
EPOCH          = 64

def one_hot_embedding (y, length):
  out = torch.zeros(length)
  out[y] = 1.0

def q_sel (self):
  for index in range(NUM_HIDDEN):
    self.sel[index] = torch.sigmoid(self.w_x[index] * self.x[index] + self.w_h[index] * self.h[index])

def q_layer (self):
  q_sel(self)

def fw_prop (self):
  q_layer(self)

dataset_train = datasets.MNIST(
    '~/mnist', 
    train=True, 
    download=True, 
    transform=transforms.ToTensor())

dataloader_train = utils.data.DataLoader(dataset_train,
                                          batch_size=BATCH_SIZE,
                                          shuffle=True,
                                          num_workers=4)

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        
        # Gate-Weight
        self.w_x   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.w_h   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
    
        # Gate-Selector
        self.sel   = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]
        
        # Input Vector
        self.x     = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]

        # Output Vector
        self.h     = [torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]

    def forward(self):
      fw_prop(self)

model = Model()
model.cuda()

print(model.parameters())
for name, param in model.named_parameters():
    if param.device.type != 'cuda':
        print('param {}, not on GPU'.format(name))

for epoch in range(EPOCH):
  for x, t in dataloader_train:
    y = one_hot_embedding(t, NUM_CLASS)
    for time in range(TIME_STEPS):
     
      model.x[0] = x[0][0][time] 
      model.h[0] = torch.zeros(NUM_INPUT)
      model.zero_grad()
      out = model()
</code></pre>
<p>makes error of;</p>
<pre><code class="lang-python">  File "test.py", line 24, in q_sel
    self.sel[index] = torch.sigmoid(self.w_x[index] * self.x[index] + self.w_h[index] * self.h[index])
RuntimeError: expected device cuda:0 and dtype Float but got device cpu and dtype Float
</code></pre>
<p>So, lack of sending data makes the error, indeed.</p>

ptrblck: <p><code>sel</code>, <code>x</code> and <code>h</code> are not wrapped in <code>nn.Parameter</code> and <code>nn.ParameterList</code>,<br>
Have a look at my <a href="https://discuss.pytorch.org/t/runtimeerror-expected-device-cpu-and-dtype-float-but-got-device-cuda-0-and-dtype-float/54783/8">code snippet</a>.</p>

111137: <p>Code does;</p>
<pre><code class="lang-python">      model.x[0] = x[0][0][time] 
      model.h[0] = torch.zeros(NUM_INPUT)
</code></pre>
<p>self.sel , self.x , self.h are not parameters.<br>
So, I think wrapping as parameter does not allow substitution.<br>
It makes an error of;</p>
<pre><code class="lang-python">in register_parameter
    .format(torch.typename(param), name))
TypeError: cannot assign 'torch.FloatTensor' object to parameter '0' (torch.nn.Parameter or None required)
</code></pre>
<p>I triied to do;</p>
<pre><code class="lang-python">        self.sel   = [torch.zeros(NUM_INPUT, device='cuda') for _ in range(NUM_HIDDEN)]
        self.x     = [torch.zeros(NUM_INPUT, device='cuda') for _ in range(NUM_HIDDEN)]
        self.h     = [torch.zeros(NUM_INPUT, device='cuda') for _ in range(NUM_HIDDEN)]
</code></pre>
<p>In training loop:</p>
<pre><code class="lang-python">      model.x[0] = torch.cuda.FloatTensor(x_)
      model.h[0] = torch.cuda.FloatTensor(torch.zeros(NUM_INPUT))
</code></pre>
<p>Makes segmentation fault</p>

ptrblck: <p>Thanks for the information. I’ve missed that these tensors should not require gradients.<br>
In that case, you could register them as buffers using:</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.w_x   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.w_h   = nn.ParameterList([nn.Parameter(torch.randn(NUM_INPUT)) for _ in range(NUM_HIDDEN)])
        self.register_buffer('sel', torch.stack([torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]))
        self.register_buffer('x', torch.stack([torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]))
        self.register_buffer('h', torch.stack([torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]))
</code></pre>

111137: <p>I do it and makes;</p>
<pre><code class="lang-python">model.x[0] = torch.cuda.FloatTensor(s)
</code></pre>
<p>Then segmentation fault is occurred on the line.</p>