Getting TypeError mismatch for CPU and GPU despite having pushed model and data to GPU

username: seankala
cooked: <p>Hello. I’m trying to run my model with some data and am getting the following error:</p>
<pre><code class="lang-auto">TypeError: expected Variable[CPUType] (got torch.cuda.FloatTensor)
</code></pre>
<p>I’ve checked some of the answers here and it seemed that I hadn’t pushed my model onto the device yet.</p>
<p>However, I checked the code and I have in fact done that, and I even explicitly pushed it onto the device in the Python Debugger interactive shell and am still getting the same error.</p>
<p>The code is as following:</p>
<pre><code class="lang-auto">
### Module: main.py
def main():
    config = get_args()
    dataset = Data(config)
    model = GCN(config, dataset.num_features, dataset.num_classes)
    trainer = Trainer(config, model, dataset)

    if torch.cuda.is_available():
        model = model.to('cuda') # I've double checked that torch.cuda.is_available() returns True.

    trainer.train()


### Module solver.py
class Trainer():
    def __init__(self, config, model, dataset):
        self.config = config
        self.num_epochs = self.config.num_epochs
        self.model = model
        self.dataset = dataset

        self.features = self.dataset.features
        self.adj_hat = self.dataset.adj_hat

    def train(self):
        self.model.train()

        optimizer = get_optimizer(self.config, self.model)
        loss_train = nn.NLLLoss()

        for epoch in range(self.num_epochs):
            optimizer.zero_grad()
            output = self.model(self.features, self.adj_hat)


### Module: models.py
class GraphConv(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.weight_mat = nn.Linear(in_features=in_features, out_features=out_features)

    def forward(self, x, adj_mat):
        weight_prod = torch.DoubleTensor(self.weight_mat(x))
        output = torch.matmul(adj_mat, weight_prod)

       return output


class GCN(nn.Module):
    def __init__(self, config, num_features, num_classes):
        super().__init__()
        self.config = config
        self.num_hidden = self.config.num_hidden
        self.num_classes = num_classes
        self.num_features = num_features
        self.p = self.config.dropout_rate

        self.graphconv1 = GraphConv(in_features=self.num_features, out_features=self.num_hidden)
        self.graphconv2 = GraphConv(in_features=self.num_hidden, out_features=self.num_clases)

    def forward(self, x, adj_hat):
        x = F.relu(self.graphconv1(x, adj_hat))
        x = F.dropout(input=x, p=self.p, training=self.training)
        output = F.softmax(self.graphconv2(x, adj_hat), dim=1)

        return output
</code></pre>
<p>The specific line of code that’s triggering the error is the <code>x = F.relu(self.graphconv1(x, adj_hat))</code> inside the GCN model. I don’t understand because if I put <code>self.model</code> on the device, shouldn’t that take care of this issue?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>You are creating new tensors on the CPU in this line of code:</p>
<pre><code class="lang-python">weight_prod = torch.DoubleTensor(self.weight_mat(x))
</code></pre>
<p>If you want to change the data type, use <code>out = out.double()</code> instead and make sure it’s the right type for all further calculations.</p>

username: helloworld2021
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, ptrblck. Could you please explain it in detail? Since the error is “expected Variable[CPUType]”, while your comment is “creating new tensors on the CPU”. In addition, I don’t understand why we need to modify the code after setting <code>model = model.to('cuda')</code></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="helloworld2021" data-post="4" data-topic="72523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/a183cd/48.png" class="avatar"> helloworld2021:</div>
<blockquote>
<p>Could you please explain it in detail? Since the error is “expected Variable[CPUType]”, while your comment is “creating new tensors on the CPU”.</p>
</blockquote>
</aside>
<p>The error message tries to be helpful in claiming which type is expected and which types were found.<br>
However, this expectation could be a guess in case both types would be valid for this operation and, if I’m not mistaken, the error message uses the parameter as the expected type and the input as the wrong one (as done here), since this issue is more common.</p>
<aside class="quote no-group" data-username="helloworld2021" data-post="4" data-topic="72523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/a183cd/48.png" class="avatar"> helloworld2021:</div>
<blockquote>
<p>In addition, I don’t understand why we need to modify the code after setting <code>model = model.to('cuda')</code></p>
</blockquote>
</aside>
<p>In this case, each forward pass creates new tensors in the <code>forward</code> method without using a device agnostic approach:</p>
<pre data-code-wrap="python"><code class="lang-python">    def forward(self, x, adj_mat):
        weight_prod = torch.DoubleTensor(self.weight_mat(x))
</code></pre>
<p>Since no <code>device</code> attribute was used, the new <code>weight_prod</code> tensors will be created on the CPU by default.<br>
Note that this approach would also detach <code>weight_prod</code> from the computation graph, so I would stick to my suggestion in using <code>.double()</code> if the <code>dtype</code> should be changed.</p>

username: helloworld2021
cooked: <p>Thank you for your help. Based on previous study and your comments,  if I’m not mistaken, the error is due to tensor transferring from cpu to gpu ( <code>model = model.to('cuda')</code>). After this operation, <code>self.weight_mat(x)</code> is on gpu but the operation <code>torch.DoubleTensor(self.weight_mat(x))</code> can only be done on cpu.</p>
<p>However, I don’t fully understand your solution. Since current error is from <code>weight_prod = torch.DoubleTensor(self.weight_mat(x))</code>, why you suggest operations on other variables <code>out = out.double()</code>?</p>
<p>How about just set</p>
<pre><code class="lang-auto">weight_prod = torch.nn.parameter(self.weight_mat(x))
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="helloworld2021" data-post="6" data-topic="72523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/a183cd/48.png" class="avatar"> helloworld2021:</div>
<blockquote>
<p>but the operation <code>torch.DoubleTensor(self.weight_mat(x))</code> can only be done on cpu.</p>
</blockquote>
</aside>
<p>This operation is explicitly creating a CPUTensor, so it’s not a limitation of where this operation can be executed, but what it’s used for.</p>
<aside class="quote no-group" data-username="helloworld2021" data-post="6" data-topic="72523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/a183cd/48.png" class="avatar"> helloworld2021:</div>
<blockquote>
<p>However, I don’t fully understand your solution.</p>
</blockquote>
</aside>
<p>What the user tried to do seemed to be a transformation of the <code>float32</code> tensor to a <code>float64</code> tensor (you could ask him to double check).<br>
This can be easily done via the <code>tensor.double()</code> operation, which neither will change the device (no device mismatch) nor will it detach the tensor.<br>
Your approach creates a new parameter (which won’t be optimized, as it’s depending on the input <code>x</code>, is recreated in each iteration, and is thus unknown to the optimizer), which will also detach the operation from the computation graph and won’t change the <code>dtype</code> to <code>float64</code>, which seems to be the original use case.</p>

username: helloworld2021
cooked: <p>Thank you for your reply. If I have to build some tensor in <code>forward</code> function, how could I avoid the conflict between cpu tensor and gpu tensor after setting <code>model.cuda()</code>?</p>
<p>In this <a href="https://discuss.pytorch.org/t/dynamic-parameter-declaration-in-forward-function/427/2">thread</a>, apaszke just recommends to use <code>nn.parameter</code>. However, in the discussion above, you said</p>
<pre><code class="lang-auto">creates a new parameter (which won’t be optimized, as it’s depending on the input x, is recreated in each iteration, and is thus unknown to the optimizer), which will also detach the operation from the computation graph
</code></pre>
<p>For me, a beginner on Pytorch, it’s somehow confusing. I can understand your comments on the effects of <code>nn.parameter</code> in <code>forward</code> function (optimizer can not work on it), then what is the right way to build a tensor in the <code>forward</code> function where the tensor is related with both the weight parameter and the output in the same time?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>If you want to create a trainable parameter or a tensor, which should be registered in the module during its initialization, you could use an <code>nn.Parameter</code> or register the tensor via <code>self.register_buffer</code> in the <code>__init__</code> method.<br>
This will make sure to transfer the tensors to the appropriate device when <code>model.to()</code> is called.<br>
However, if you want to create a new tensor in the <code>forward</code> method (which is different than the original question), you could reuse the <code>.device</code> attribute of a known parameter or the input:</p>
<pre><code class="lang-python">def forward(self, x):
    my_new_tensor = torch.randn(1, device=x.device)
    x = x + my_new_tensor
    return x
</code></pre>

username: helloworld2021
cooked: <p>I’m grateful for your help. Now I think I should avoid using <code>nn.parameter</code> in the <code>forward</code> function and use your method instead.</p>
<p>Thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=9" title=":tada:" class="emoji" alt=":tada:"> <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=9" title=":tada:" class="emoji" alt=":tada:"> <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=9" title=":tada:" class="emoji" alt=":tada:"></p>