RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)
❓ Questions and Help
Before asking:

Try to find answers to your questions in the Lightning Forum!
Search for similar issues.
Search the docs.


What is your question?
Hi,
I'm using pytorch lightning 1.1.0 to train my model. However, I got the following error:
  File "./scripts/run_model.py", line 137, in <module>
    trainer.fit(model, train_dataloader=train_loader, val_dataloaders=val_loader)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 470, in fit
    results = self.accelerator_backend.train()
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 66, in train
    results = self.train_or_test()
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 65, in train_or_test
    results = self.trainer.train()
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 492, in train
    self.run_sanity_check(self.get_model())
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 690, in run_sanity_check
    _, eval_results = self.run_evaluation(test_mode=False, max_batches=self.num_sanity_val_batches)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 606, in run_evaluation
    output = self.evaluation_loop.evaluation_step(test_mode, batch, batch_idx, dataloader_idx)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 178, in evaluation_step
    output = self.trainer.accelerator_backend.validation_step(args)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 84, in validation_step
    return self._step(self.trainer.model.validation_step, args)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 74, in _step
    output = model_step(*args)
  File "/export/home/cma/explainable_DTD_model/KG1/GATConv_model_pytorchlightning/scripts/model.py", line 275, in validation_step
    pred_y = self.forward(x_n_id, adjs, link, n_id, return_attention=False)
  File "/export/home/cma/explainable_DTD_model/KG1/GATConv_model_pytorchlightning/scripts/model.py", line 189, in forward
    x = self.convs[i]((x, x_target), edge_index)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/export/home/cma/explainable_DTD_model/KG1/GATConv_model_pytorchlightning/scripts/model.py", line 91, in forward
    x_l = self.lin_l(x_l).view(-1, H, C)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/cma/work/miniconda3/envs/mypersonal_env/lib/python3.8/site-packages/torch/nn/functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

I understand this error is because the tensors or layers might not in the same device. But I have no idea how to fix it. Since the tutorial of pytorch lightning multi-gpu page suggests to delete all .cuda() or .to() calls. So I think all layers and tensors should be automatically moved to the appropriate device.
I didn't get this error when I use pytorch where I used some .to() functions.
Could you please give me some suggestions about how to fix this error. My __init__ part and __forward__ part code is below
Code
class GAT(pl.LightningModule):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers, dropout_p, batch_set, idx_map, feat, num_head = 8, lr=0.005):
        super(GAT, self).__init__()

        self.num_layers = num_layers
        self.dropout_p = dropout_p
        self.lr = lr
        self.save_hyperparameters('in_channels','hidden_channels','out_channels','num_layers','dropout_p','num_head','lr')

        self.convs = torch.nn.ModuleList()
        self.convs.append(GATConv(in_channels, hidden_channels, heads=num_head, dropout=0, concat=False))
        for _ in range(num_layers - 1):
            self.convs.append(GATConv(hidden_channels, hidden_channels, heads=num_head, dropout=0, concat=False))
        self.lin = torch.nn.Linear(hidden_channels*2,out_channels)

    def forward(self, x, adjs, link, n_id, return_attention=False):

        for i, (edge_index, size) in enumerate(adjs):
            x_target = x[:size[1]]
            x = self.convs[i]((x, x_target), edge_index)
            x = F.elu(x)
            x = F.dropout(x, p=self.dropout_p, training=self.training)

        if link.shape[0]==1:
            x = torch.cat([x[[torch.where(n_id==i)[0][0] for i in link[:,0]]].view(1,-1),x[[torch.where(n_id==i)[0][0] for i in link[:,1]]].view(1,-1)], dim=1)
        else:
            x = torch.cat([x[[torch.where(n_id==i)[0][0] for i in link[:,0]],:],x[[torch.where(n_id==i)[0][0] for i in link[:,1]],:]], dim=1)

        x = self.lin(x).squeeze(1)   

        return x
What have you tried?
I tried to add some .to() in init to move the layers to cuda device but it doesn't work.
What's your environment?

OS: Linux
Packaging: conda
Version: pytorch_lightning 1.1.0