RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 0 does not equal 2 (while checking arguments for cudnn_convolution)

username: 1414b35e42c77e0a57dd
cooked: <p>Hi, all<br>
Why I should use same gpu number when I trained the model?<br>
If not, this error occurs.<br>
<code>RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 0 does not equal 2 (while checking arguments for cudnn_convolution)</code></p>

username: ptrblck
cooked: <p>The input data and parameters of a module need to be located on the same GPU, since the calculation is performed on this device.<br>
In your current code it seems your input is on GPU0, while the <code>weight</code> parameter is on GPU2.<br>
Make sure to push the input to the same device where the model is located.</p>

username: pix_1
cooked: <p>Just want to add something here:<br>
I ran into the same problem when I resumed a training.</p>
<blockquote>
<p>“When you call torch.load() on a file which contains GPU tensors, those tensors will be loaded to GPU by default.”</p>
</blockquote>
<p>In my case, I run trainings on different GPUs on a multi GPU-system. So from time to time my GPU changes.</p>
<p>So, to avoid this loading issues:</p>
<blockquote>
<p>torch.load(checkpoint_file, map_location=‘cpu’)</p>
</blockquote>
<p>Later you can push everything on the right device.</p>

username: zshn25
cooked: <p>I have the same error with DistributedDataParallel. I am sending the model to DDP as follows</p>
<pre><code class="lang-auto">models = DDP(models.cuda(), device_ids=[rank],
                            broadcast_buffers=False, find_unused_parameters=True)
</code></pre>
<p>and I am sending the inputs to the same rank as well. This error doesn’t occur while using DataParallel though</p>

username: ptrblck
cooked: <p>Could you post the code snippet which shows how you are transferring the input tensors to the right device?<br>
I guess you might transfer it to the default device instead of the currently used one.</p>

username: zshn25
cooked: <p>Sending the model to the <code>rank</code> instead of <code>cuda</code> solved it for me. Thanks</p>

username: wtliao
cooked: <p>I was suffering the same issue. But it seems  not to be caused by GPU0 problem. Because if you use CUDA_VISIBLE_DEVICES=1,3,4 to specify the used GPUS, they are reindexed in the pytorch process as range(<span class="hashtag">#available_GPUS</span>) (same as in tensorflow).<br>
After many tries, I find the problem in the definition of a model and solve as fellows. I originally defined my model as :</p>
<pre><code class="lang-auto">self.features, self.layer4 = load_resnet()
            self.roi_fmap = self._roi_map

def forward(self, x):
   ...
   return self.roi_fmap(x)
....
# for convenient, I wrap the operation here
def _roi_map(self, features):
        return self.layer4(features).mean(3).mean(2) 
</code></pre>
<p>Then I solve the problem by modifying</p>
<pre><code class="lang-auto">self.features, self.layer4 = load_resnet()
            self.roi_fmap = self.layer4

def forward(self, x):
   ...
  out = self.roi_fmap(x)
   return out.mean(3).mean(2)
</code></pre>
<p>I guest that when I use a function  _roi_map() to wrap a module self.layer4(features).mean(3).mean(2), pytorch seems  not to assige this part to different gpus by using replicas = nn.parallel.replicate(model, devices=list(range(num_gpus))).<br>
Any layer/module/model should be exposed in the def <strong>init</strong>() part of a  nn.Module class. pytorch seems only search relevant layer/module/model here and  not to dive into any functions within or outside the class definition.</p>
<p>I am not sure whether my analysis correct. If you know more please correct me. Thanks a lot.</p>

username: Xiaoyu_Song
cooked: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a>, I use the code below:</p>
<pre><code class="lang-auto">device = torch.device("cuda", gpu_ids[0]) if args.gpu_ids is not None else torch.device("cuda")
x = imgs.to(device = device, dtype = torch.float32)
net = torch.nn.DataParallel(net, device_ids = gpu_ids, output_device = device).to(device=device)
</code></pre>
<p>I got the same error. Could you help?<br>
If I comment out the DataParallel part, the code works fine</p>

username: ptrblck
cooked: <p>Could you post your model definition and the complete stack trace?<br>
I guess you might create tensors in a specific device inside your model or are using tensors instead of buffers of parameters.</p>

username: Xiaoyu_Song
cooked: <p>I use a pretraned network from github.</p>
<pre><code class="lang-auto">net = WSDAN(num_classes=2, M=8, net="efficientnet-b3", pretrained=False)
ckpt = torch.load("./pretrained/abcd.pth", map_location="cpu")
net.load_state_dict(ckpt["state_dict"])
</code></pre>