RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)

username: nbansal90
cooked: <p>Hi All,<br>
I am using <code>torch.nn.DataParallel</code> for the model in multiple gpu setup (4 GPUs)as defined below and find that there is some mismatch between the <code>input</code> and <code>weight</code>. I am using the following <code>Module</code>  for a <code>Decoder</code> kind of a network.</p>
<pre><code class="lang-auto">class DepthDecoder(nn.Module):
    def __init__(self, num_ch_enc, scales=range(4), num_output_channels=1, max_depth=10, use_skips=True, use_bn=False):
        super(DepthDecoder, self).__init__()

        self.num_output_channels = num_output_channels
        self.max_depth = max_depth
        self.use_skips = use_skips
        self.upsample_mode = 'nearest'
        self.scales = scales

        self.num_ch_enc = num_ch_enc
        self.num_ch_dec = np.array([16, 32, 64, 128, 256])

        # decoder
        self.convs = OrderedDict()
        for i in range(4, -1, -1):
            # upconv_0
            num_ch_in = self.num_ch_enc[-1] if i == 4 else self.num_ch_dec[i + 1]
            num_ch_out = self.num_ch_dec[i]
            if use_bn:
                self.convs[("upconv", i, 0)] = ConvBlock_bn(num_ch_in, num_ch_out)
            else:
                self.convs[("upconv", i, 0)] = ConvBlock(num_ch_in, num_ch_out)

            # upconv_1
            num_ch_in = self.num_ch_dec[i]
            if self.use_skips and i &gt; 0:
                num_ch_in += self.num_ch_enc[i - 1]
            num_ch_out = self.num_ch_dec[i]
            if use_bn:
                self.convs[("upconv", i, 1)] = ConvBlock_bn(num_ch_in, num_ch_out)
            else:
                self.convs[("upconv", i, 1)] = ConvBlock(num_ch_in, num_ch_out)

        for s in self.scales:
            # self.convs[("dispconv", s)] = Conv3x3(self.num_ch_dec[s], self.num_output_channels)
            self.convs[("depthconv", s)] = Conv3x3(self.num_ch_dec[s], self.num_output_channels)

        self.decoder = nn.ModuleList(list(self.convs.values()))
        self.sigmoid = nn.Sigmoid()

def forward(self, input_features):
        self.outputs = {}

        # decoder
        x = input_features[-1]
        for i in range(4, -1, -1):
            x = self.convs[("upconv", i, 0)](x)
            x = [upsample(x)]
            if self.use_skips and i &gt; 0:
                x += [input_features[i - 1]]
            x = torch.cat(x, 1)
            x = self.convs[("upconv", i, 1)](x)
            if i in self.scales:
                # self.outputs[("disp", i)] = self.sigmoid(self.convs[("dispconv", i)](x))
                self.outputs[("depth", i)] = self.sigmoid(self.convs[("depthconv", i)](x)) * self.max_depth

        return self.outputs
</code></pre>
<p><code>ConvBlock</code> and <code>Conv3x3</code> are just simple convultional class performing 2D conv.</p>
<pre><code class="lang-auto">class ConvBlock(nn.Module):
    """Layer to perform a convolution followed by ELU
    """
    def __init__(self, in_channels, out_channels):
        super(ConvBlock, self).__init__()

        self.conv = Conv3x3(in_channels, out_channels)
        self.nonlin = nn.ELU(inplace=True)

    def forward(self, x):
        out = self.conv(x)
        out = self.nonlin(out)
        return out
</code></pre>
<p>I have seen similar post with the similar error, and what I get is this is something to do with <code>redefinition</code> of a function in <code>init</code> function, and is about putting as much as functionality in <code>forward</code> function. But I was quite not able to pin-point, what exactly was causing this issue in my case.</p>
<p>Any suggestion would be really helpful!</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet as the current code doesn’t define the <code>num_ch_enc</code> (shape and values), the <code>upsample</code> usage as well as the input shape?</p>

username: nbansal90
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, Please find my response inline:</p>
<ol>
<li>
<p>define the <code>num_ch_enc</code> (shape and values)<br>
-------&gt; Here in this case, <code>num_ch_enc</code> defines the number of channel which is a <code>&lt;class 'numpy.ndarray'&gt;</code> containing the value  <code> [64  64 128 256 512]</code> of shape <code>(5,)</code></p>
</li>
<li>
<p>Input shape to <code>forward</code> function<br>
-------&gt; Input to <code>forward</code> function is <code>list</code> of <code>Len</code> 5, where each element is a tensor of size:</p>
</li>
</ol>
<pre><code class="lang-auto">torch.Size([8, 64, 240, 320])
torch.Size([8, 64, 120, 160])
torch.Size([8, 128, 60, 80])
torch.Size([8, 256, 30, 40])
torch.Size([8, 512, 15, 20])
</code></pre>
<p>Where 8 is the <code>Batch Size</code> (which rightly reduces to 2, when I use multiple GPUs(4) with Dataparallel)</p>
<ol start="3">
<li>Definition of <code>Upsample</code>  and <code>Conv3x3</code>
</li>
</ol>
<pre><code class="lang-auto">def upsample(x):
    """Upsample input tensor by a factor of 2
    """
    return F.interpolate(x, scale_factor=2, mode="nearest")
</code></pre>
<pre><code class="lang-auto">class Conv3x3(nn.Module):
    """Layer to pad and convolve input
    """
    def __init__(self, in_channels, out_channels, use_refl=True):
        super(Conv3x3, self).__init__()

        if use_refl:
            self.pad = nn.ReflectionPad2d(1)
        else:
            self.pad = nn.ZeroPad2d(1)
        self.conv = nn.Conv2d(int(in_channels), int(out_channels), 3)

    def forward(self, x):
        out = self.pad(x)
        out = self.conv(out)
        return out

</code></pre>
<p>Data Parallel part goes without any hassle for <code>Encoder</code> Part but it fails and gives this error, when it goes through the <code>DepthDecoder</code> network. (Output of <code>Encoder</code> goes-to <code>DepthDecoder</code> class specified before.</p>

username: ptrblck
cooked: <p>Thanks for the update.<br>
The issue is raised, because <code>self.convs</code> is using and <code>OrderedDict</code> instead of an <code>nn.ModuleDict</code>, which will not properly register these modules.<br>
Change it to the latter and adapt the indexing, as <code>nn.ModuleDict</code> expects strings (e.g. to <code>"upconv{}{}".format(i, 0)</code>).<br>
Also, note that <code>nn.DataParallel</code> will split each element in your list in <code>dim0</code>, so that each GPU will receive a list of 5 tensors in the shape:</p>
<pre><code class="lang-python">torch.Size([1, 64, 240, 320])
torch.Size([1, 64, 120, 160])
torch.Size([1, 128, 60, 80])
torch.Size([1, 256, 30, 40])
torch.Size([1, 512, 15, 20])
</code></pre>
<p>using 8 GPUs.</p>

username: nbansal90
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> that worked like a charm!!</p>