RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1

Shubhankar: <p>I have 4xP100s and saved an initial checkpoint after <code>nn.DataParallel(model)</code> but when I tried to load it in another model I got an error</p>
<pre><code class="lang-auto">RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/gpfs/alpine/world-shared/gen011/shubhankar/summitdev/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/gpfs/alpine/world-shared/gen011/shubhankar/summitdev/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/gpfs/alpine/world-shared/gen011/shubhankar/summitdev/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 146, in forward
    "them on device: {}".format(self.src_device_obj, t.device))
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1
</code></pre>

aauker: <p>It would be useful to have either your code (if it’s simple), or a minimal example to help understand why you get this message.  Just some background, if I remember correctly, the actual model parameters must be stored on the default GPU (id=0), though calculations are carried out in parallel.</p>
<p>A guess I can provide now is this: when loading a parallelized checkpoint, you must initialize the model including any .cuda() calls before loading the checkpoint.  It might be you’ve done this after.</p>
<p>A scheme like this should work:</p>
<pre><code class="lang-auto">model = SomeModel()
model = nn.DataParallel( model ).cuda()
model.load_state_dict(checkpoint)
</code></pre>

Shubhankar: <p>I am just following this <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">doc</a>. I mean as far as loading the model is concerned. Other details are the regular load, train, test loops.</p>

aauker: <p>I actually remember having the same exact issue, and following that same doc.  Could you try the scheme I gave?  In either case post the part of your script where you initialize the model.</p>

Shubhankar: <pre><code class="lang-auto">     network = models.resnet18(pretrained=False)
     network.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
     network.fc = nn.Linear(512, out_features=27)
     loader = torch.utils.data.DataLoader(train_set, batch_size=run.batch_size, 
     num_workers=run.num_workers, shuffle=False, pin_memory=False)
     with torch.no_grad():
         test_loader = torch.utils.data.DataLoader(test_set, batch_size=run.batch_size, 
                               num_workers=run.num_workers, pin_memory=False)
   
    network.to(device)
    optimizer = optim.AdamW(network.parameters(), lr=run.lr, weight_decay=run.weight_decay)
   
    with open('epoch-0.pth', 'rb') as f:
        checkpoint = torch.load(f)
        network.module.load_state_dict(checkpoint['model_state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])    
    network = nn.DataParallel(network)
    network.fc = nn.Linear(512, out_features=3)
</code></pre>

Shubhankar: <p>I tried appending .cuda() to nn.DataParallel(model) but didn’t work.</p>

aauker: <p>I think the order matters in this case. Did you try wrapping in nn.DataParallel before either .to(device) or .cuda() and before leading the checkpoint?</p>

Lucian_Mircea_Sasu: <p>Is there any solution to this issue? I encountered the same situation.</p>

Shubhankar: <p>What’s your issue. I did figure out the solution.</p>

lmsasu: <p>Hi <a class="mention" href="/u/shubhankar">@Shubhankar</a>, sorry for delay. Right now I cannot have a minimum working example, as I rewrote the code and the bug disappeared.</p>

Leigh_Davis: <p>Hi <a class="mention" href="/u/shubhankar">@Shubhankar</a>, can you please share your solution with us as well? I am in the same boat; having exactly the same issue. Thanks!</p>

Shubhankar: <p>What issue are you facing?</p>

Vivek_Kumar: <p>module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:2</p>
<p>I am also facing same issue any solution?</p>

ptrblck: <p>Could you post a minimal executable code snippet to reproduce the issue, please?</p>

Bhavay_Malhotra: <p>Hey everyone, I’m facing the same issue. I trained a resnet model on cuda:0 and I want to load it on cuda:1. I have the model stored in a .t7 file format, soo I don’t know if the whole model was saved or its parameters. How can I load the model on other gpu other than that on which it was trained on?</p>

AlphaBetaGamma96: <p><a class="mention" href="/u/bhavay_malhotra">@Bhavay_Malhotra</a> If your <code>model</code> is currently on <code>cuda:0</code> you can move it to <code>cuda:1</code> via <code>model = model.to('cuda:1')</code> or in general by using the <code>.to</code> method.</p>

Bhavay_Malhotra: <p>Hi, thanks for the reply. Actually the model was originally trained on cuda:0 and all its parameters and buffer is stored on cuda:0 . If i change the model to load on cuda:1, will it cause any error??</p>

AlphaBetaGamma96: <p>It should be fine, move it across and see if it runs.</p>

Bhavay_Malhotra: <p>Hey <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> , I changed every device to cuda:1 in every file i import, still the code gives me the above error that model parameters and buffer are on cuda:0.</p>

AlphaBetaGamma96: <p>Can you print the exact error message? Also, are you using an optimizer here? Because the <code>optim</code> might have the parameters and buffers on the original device too.</p>