Tensors are in multiple cuda devices

hamedB: <p>Hi<br>
In various post, I have seen the comments on how to utilize<br>
<code>torch.nn.DataParallel</code> to use multiple GPUs. Also, my code follow the template provided in this <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="noopener nofollow ugc">link</a>. However, during the convolutional operation, I face such error.</p>
<p><code>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!</code></p>
<p>Could you please help me out with this error?</p>

AlphaBetaGamma96: <p>Can you post a minimal reproducible example?</p>

hamedB: <p>Thank you for your reply.</p>
<p>Actually, the code is right now complex, so if I want to provide a minimal example of that, it might not be my case. However, some important parts of my code are as below.</p>
<p>In here, model is Alexnet pre-trained by Image-Net.</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = torch.nn.DataParallel(model)
model = model.to(device)
</code></pre>
<p>During the passing an image through the layers, the featuremaps produced by each filter are multiply by a weight called <code>layer_landas</code> like bellow.<br>
<code>x</code> is input image. The x tensor is also transferred to gpu by <code>.to(device)</code> command before passing through each module of Alexnet.</p>
<pre><code class="lang-auto">layer_landas = layer_landas.to(device)
out_fm = self.conv_layer(x)
out_fm = torch.mul(layer_landas, out_fm)
</code></pre>
<p>The error is:</p>
<pre><code class="lang-auto">File "/project/Code/cnn.py", line 61, in __call__
out_fm = torch.mul(layer_landas, out_fm)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
</code></pre>

ptrblck: <p>I guess you’ve defined <code>device</code> once globally and are now reusing it in the <code>forward</code> in:</p>
<pre><code class="lang-python">layer_landas = layer_landas.to(device)
</code></pre>
<p>If so, this would cause the issue since <code>nn.DataParallel</code> creates copies of the model on all specified devices, while you are moving <code>laber_landas</code> to <code>device</code> explicitly.<br>
Use <code>layer_landas = layer_landas.to(x.device)</code> and it might work as the input tensor’s device would be used instead.</p>

hamedB: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>  for replying.</p>
<p>I followed your suggestion, but the same error was throwed in another line when I want to pass the feature map from previous layer into the next conv layer.</p>
<pre><code class="lang-auto">out_fm = self.conv_layer(x)
</code></pre>
<p>The error is:</p>
<pre><code class="lang-auto">out_fm = self.conv_layer(x)
   File "/opt/conda/envs/pytorch1_10/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
     return forward_call(*input, **kwargs)
   File "/opt/conda/envs/pytorch1_10/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 446, in forward
2    return self._conv_forward(input, self.weight, self.bias)
   File "/opt/conda/envs/pytorch1_10/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 442, in _conv_forward
     return F.conv2d(input, weight, bias, self.stride,
 RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper___conv_depthwise2d)
</code></pre>
<p>Some part of my code.</p>
<p>main function:</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = torch.nn.DataParallel(model)
model = model.to(device)
</code></pre>
<p>call function in model class (inheres nn.module): in this function, I pass the featuremaps through each module one by one. the feature_extractor, move on each module of feature part pf the CNN. you can see its body in the next code snippet.</p>
<pre><code class="lang-auto">    def __call__(self, data):
        featuremaps = []
        for name, module in self.modified_cnn._modules.items():
            if name == 'features':
                featuremaps, x = self.feature_extractor(data, self.modified_cnn.features, self.landa)
            elif "avgpool" in name.lower():
                x = module(x)
                x = x.view(x.size(0), -1)
            else:
                x = module(x)

        return featuremaps, x
</code></pre>
<p>class FeatureExtractor:</p>
<p>in here, when I reach to a <code>custom_convlayer</code> modul, pass the featuemap x and layer_landas to the module. The next code snippet shows a part of this module.</p>
<pre><code class="lang-auto">def __call__(self, x, model, graph_landas=None):
    self.gradients = []
    featuremaps = []
    for name, module in model._modules.items():
        if type(module) == custom_convlayer:
            x = module((x, layer_landas))
        elif type(module) == torch.nn.modules.activation.ReLU:
            xx = torch.clone(x)
            x = module(xx.to(device))
            del xx
        else:
            x = module(x.to(device))

    return featuremaps, x
</code></pre>
<p>class custom_convlayer<br>
In here when I want to pass featuremap into the conv layer, that error is throwed.</p>
<pre><code class="lang-auto">
    def __call__(self, data):

        x = data[0]
        layer_landas = data[1]

        out_fm = self.conv_layer(x)
</code></pre>

ptrblck: <p>I guess you are hitting the same issue again.<br>
Don’t use the global <code>device</code> attribute, as <code>nn.DataParallel</code> will create copies on all passed GPUs.<br>
E.g. if you are passing 2 GPU ids to <code>nn.DataParallel</code> each GPU will get a clone of the model and the input data will also be split in the batch dimension and pushed to the corresponding device.<br>
The model on <code>cuda:0</code> will then get the input tensor on <code>cuda:0</code> and the clone on <code>cuda:1</code> will get the input tensor on <code>cuda:1</code>.<br>
If you are now creating new tensors inside the model with <code>device='cuda:0'</code> it will raise a device mismatch, so use the <code>.device</code> attribute of the input or any registered parameter.</p>
<p>Also, don’t use the <code>__call__</code> method, but implement the <code>forward</code> since the <code>__call__</code> is used internally in <code>nn.Module</code>s.</p>

hamedB: <p>Ok. As I understud, I shouldn’t transfer manually the input into the device and since I use nn.DataParallel, such transformation is done internally. So I comment out that part of my code where I was transferring the input image into the device. However, the issue remained unchange, and the error the previous error is throwed.</p>
<p>part of the code that I pass input image into the model.</p>
<pre><code class="lang-auto">for inputs, labels, paths in data_loaders['train']:
      optimizer.zero_grad()
      #inputs = inputs.to(device)
      _, modified_output = modified_model((inputs, settings.class_index))
</code></pre>

hamedB: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
The problem was solved. I comment out anywhere that I transfer manually the input tensor into device.</p>
<p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

hamedB: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I have a follow up issue regarding this.<br>
In my code, I defined a parameter as a type <code>nn.ParameterDict</code>. During the cpu mode and gpu mode with one gpu, when I print the keys of this dictionary the output is as follow.</p>
<p><code>odict_keys(['0', '1', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '2', '3', '4', '5', '6', '7', '8', '9'])</code></p>
<p>However, when I want to use multiple gpus, this dictionary is empty while its not empty and the program throws error. The out out is as follow.</p>
<pre><code class="lang-auto">odict_keys([])
odict_keys([])
odict_keys([])
File "/project_antwerp/PathExtraction/Code/models.py", line 53, in __call__
  class_landas = graph_landas[class_index]
   File "/opt/conda/envs/pytorch1_10/lib/python3.9/site-packages/torch/nn/modules/container.py", line 586, in __getitem__
     return self._parameters[key]
 KeyError: '18'
</code></pre>
<p>I was wondering if you could please give me a help regarding this matter.</p>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce this issue, please?<br>
Without seeing code I would guess that you might be creating the <code>ParameterDict</code> object too late, i.e.after the <code>DataParallel</code> wrapping.</p>

hamedB: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  I provided a minimal case of my code that can be run. I put two print function in the class <code>FeatureExtractor</code> where I want to display the content of the parameter called <code>graph_landas</code>. Also, I create the parameter before wrapping it by nn.DataParallel.</p>
<p>Thank you in advance for your attention given to my issue.</p>
<pre><code class="lang-auto">from torch import nn
from torchvision import models
from torchvision import datasets
from torchvision import transforms
from PIL import Image
import torch
from torch import optim
</code></pre>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class FeatureExtractor():
    def __init__(self):
        self.gradients = []

    def save_gradient(self, grad):
        self.gradients.append(grad)

    def __call__(self, x, model, graph_landas=None):
        self.gradients = []
        featuremaps = []
        labels = x[1]
        x = x[0]
        print('printing landa parameter ', graph_landas)
        graph_landas = graph_landas.to(x.device)
        class_landas = None
        for class_index in [0,1,2,3,4]:
            class_index = str(class_index)
            if class_landas == None:
                class_landas = graph_landas[class_index]
            else:
                class_landas = torch.concat((class_landas, graph_landas[class_index]), 1)

        print('landa belonging to each class \n', class_landas)

        return featuremaps, x

class create_modifiedCNN(nn.Module):
    def __init__(self, model):
        super(create_modifiedCNN, self).__init__()
        self.modified_cnn = model
        self.feature_extractor = FeatureExtractor()
        num_classes = 5
        self.landa = nn.ParameterDict({
            str(class_index) : nn.Parameter(torch.ones((10,1),device=device),requires_grad=True)
            for class_index in range(num_classes)
        })

    def __call__(self, data):
        featuremaps = []
        for name, module in self.modified_cnn._modules.items():
            if name == 'features':
                featuremaps, x = self.feature_extractor(data, self.modified_cnn.features, self.landa)
            elif "avgpool" in name.lower():
                x = module(x)
                x = x.view(x.size(0), -1)
            else:
                x = module(x)
        return featuremaps, x



model = models.vgg16(pretrained=True)
out_feature = 5
in_feature = model.classifier[6].in_features
model.classifier[6] = nn.Linear(in_features=in_feature, out_features=out_feature, bias=True)
modified_model = torch.nn.DataParallel(create_modifiedCNN(model))
modified_model = modified_model.to(device)

criterion = torch.nn.CrossEntropyLoss()
optimizer_specs = [{'params': modified_model.module.landa[class_index]}
                   for class_index in modified_model.module.landa.keys()]
optimizer = optim.Adam(optimizer_specs, lr=0.1)
scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)

#### Creating data transformation
image_size = 128
data_transform = {'train': transforms.Compose([
        transforms.Resize((image_size, image_size), Image.BILINEAR),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
        'val': transforms.Compose([
            transforms.Resize((image_size, image_size), Image.BILINEAR),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),

        'test': transforms.Compose([
            transforms.Resize((image_size, image_size), Image.BILINEAR),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])
    }

### Creating dataloader
trainset = datasets.CIFAR10(root='./data', train=True, download=True, transform=data_transform['train'])
train_loader = torch.utils.data.DataLoader(trainset, batch_size=1,shuffle=True, num_workers=1)
testset = datasets.CIFAR10(root='./data', train=False,download=True, transform=data_transform['test'])
test_loader = torch.utils.data.DataLoader(testset, batch_size=1,shuffle=False, num_workers=1)
dataloaders = {'train': train_loader, 'test': test_loader}

### Feeding the model with images
for inputs, labels in dataloaders['train']:
    optimizer.zero_grad()
    class_index = labels
    _, modified_output = modified_model((inputs,class_index))
</code></pre>

ptrblck: <p>I cannot reproduce the issue as your code runs into a shape mismatch:</p>
<pre><code class="lang-python">RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x147 and 25088x4096)
</code></pre>
<p>After adding:</p>
<pre><code class="lang-python">model.classifier[0] = nn.Linear(147, 4096)
</code></pre>
<p>the code runs fine and the parameters are printed.</p>

hamedB: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for try the code.</p>
<p>Could you please tell me what is your batchsize and how many gpus you used?because when I use one gpu the code is correct, but when I use more than one gpu, I face such issue. Also, I didn’t understand where you faced the error you mentioned. Normally you should got the error in <code>FeatureExtractor</code> because I didn’t implement complete code for passing the tensor through the module. The issue happens before passing to any module.<br>
I still get the error on the parameter. I sent you a screenshot of the output.<br>
As you can see, when I print the paramter where they are created (i.e. class <code> create_modifiedCNN</code>), I can see the content. However, when I pass the image through the network, the parameter is empty.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8093de975a205c1cbd86fcc5f00ac1ea4e95da4.png" data-download-href="https://discuss.pytorch.org/uploads/default/e8093de975a205c1cbd86fcc5f00ac1ea4e95da4" title="out"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8093de975a205c1cbd86fcc5f00ac1ea4e95da4_2_690x305.png" alt="out" data-base62-sha1="x6GzzjpFhySv8JsturzmUGk7TJq" width="690" height="305" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8093de975a205c1cbd86fcc5f00ac1ea4e95da4_2_690x305.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8093de975a205c1cbd86fcc5f00ac1ea4e95da4_2_1035x457.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8093de975a205c1cbd86fcc5f00ac1ea4e95da4_2_1380x610.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">out</span><span class="informations">1463×647 195 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>