Can you combine 3d and 2d convolutions in the same network?

Khrystyna_Faryna: <p>I am trying to train a custom network in which there is a combination of 2D and 3D convolutional layers.<br>
I am getting an  RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR error when loss.backward() is computed.</p>
<blockquote>
<p>loss.backward()<br>
File “/usr/local/lib/python3.7/site-packages/torch/tensor.py”, line 166, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/usr/local/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True) # allow_unreachable flag<br>
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</p>
</blockquote>
<p>Is this because there is a problem with backward gradient computation when combining 3d and 2D conv layers or should I look for a problem somewhere else?</p>
<p>If I set torch.backends.cudnn.enabled = False nothing changes .</p>

J_Johnson: <p>Had no issues with it on an Nvidia 3090 RTX GPU with a modified UNet that has two branches, one for 2d and one for 3d here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/CerebralSeed/Hybrid-3D-UNet">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/CerebralSeed/Hybrid-3D-UNet" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/e27e965c4b5077bd889745434dae26ff1a8caf589afda7914991aa9da5ab6491/CerebralSeed/Hybrid-3D-UNet" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/CerebralSeed/Hybrid-3D-UNet" target="_blank" rel="noopener nofollow ugc">GitHub - CerebralSeed/Hybrid-3D-UNet: Model for Hybrid 3D UNet</a></h3>

  <p>Model for Hybrid 3D UNet. Contribute to CerebralSeed/Hybrid-3D-UNet development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Can you post your model?</p>

Khrystyna_Faryna: <blockquote>
<p>class ResNet(nn.Module):<br>
def <strong>init</strong>(self, dataset, depth, num_classes, bottleneck=False):<br>
super(ResNet, self).<strong>init</strong>()<br>
self.dataset = dataset</p>
<pre><code>  blocks ={18: BasicBlock, 34: BasicBlock, 50: Bottleneck, 101: Bottleneck, 152: Bottleneck, 200: Bottleneck}
  layers ={18: [2, 2, 2, 2], 34: [3, 4, 6, 3], 50: [3, 4, 6, 3], 101: [3, 4, 23, 3], 152: [3, 8, 36, 3], 200: [3, 24, 36, 3]}
  assert layers[depth], 'invalid detph for ResNet (depth should be one of 18, 34, 50, 101, 152, and 200)'

  self.inplanes = 64
  self.conv1_3d = nn.Conv3d(3, self.inplanes, kernel_size=3, stride=1, padding=1, bias=False)
  self.aveadpool1_3d = nn.AdaptiveAvgPool3d((128,128,1))
  self.conv1 = nn.Conv2d(self.inplanes, self.inplanes, kernel_size=7, stride=2, padding=3, bias=False)
  self.bn1 = nn.BatchNorm2d(64)
  self.relu = nn.ReLU(inplace=True)
  self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
  self.layer1 = self._make_layer(blocks[depth], 64, layers[depth][0])
  self.layer2 = self._make_layer(blocks[depth], 128, layers[depth][1], stride=2)
  self.dropout20 = nn.Dropout(p=0.2)
  self.layer3 = self._make_layer(blocks[depth], 256, layers[depth][2], stride=2)
  self.dropout30 = nn.Dropout(p=0.3)
  self.layer4 = self._make_layer(blocks[depth], 512, layers[depth][3], stride=2)
  self.dropout40 = nn.Dropout(p=0.4)
  

  # self.avgpool = nn.AvgPool2d(7)
  self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
  self.dropout50 = nn.Dropout(p=0.5)
  self.fc = nn.Linear(512 * blocks[depth].expansion, num_classes)
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
            m.weight.data.normal_(0, math.sqrt(2. / n))
        if isinstance(m, nn.Conv3d):
            n = m.kernel_size[0] * m.kernel_size[1] *m.kernel_size[2]* m.out_channels
            m.weight.data.normal_(0, math.sqrt(2. / n))
        elif isinstance(m, nn.BatchNorm2d):
            m.weight.data.fill_(1)
            m.bias.data.zero_()

def _make_layer(self, block, planes, blocks, stride=1):
    downsample = None
    if stride != 1 or self.inplanes != planes * block.expansion:
        downsample = nn.Sequential(
            nn.Conv2d(self.inplanes, planes * block.expansion,
                      kernel_size=1, stride=stride, bias=False),
            nn.BatchNorm2d(planes * block.expansion),
        )

    layers = []
    layers.append(block(self.inplanes, planes, stride, downsample))
    self.inplanes = planes * block.expansion
    for i in range(1, blocks):
        layers.append(block(self.inplanes, planes))

    return nn.Sequential(*layers)

def forward(self, x):
    
  x = self.conv1_3d(x)
  x = self.aveadpool1_3d(x)
  x = torch.squeeze(x, -1)
  x = self.conv1(x)
  x = self.bn1(x)
  x = self.relu(x)
  x = self.maxpool(x)

  x = self.layer1(x)
  x = self.layer2(x)
  #x = self.dropout20(x)
  x = self.layer3(x)
  #x = self.dropout30(x)
  x = self.layer4(x)
  #x = self.dropout40(x)
  
  x = self.avgpool(x)
  x = self.dropout50(x)
  x = x.view(x.size(0), -1)
  x = self.fc(x)


    return x
</code></pre>
</blockquote>

J_Johnson: <p>Your <code>self.conv1</code> layer looks like the suspect. Or at least a culprit.</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(self.inplanes, self.inplanes, kernel_size=7, stride=2, padding=3, bias=False)
</code></pre>
<p>You have size 128, 128 going in. With a stride of 2, you end up with 127/2 and that doesn’t yield an integer.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a2e32b4d21e6f40ec7ece2d879c580acb2533b1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3a2e32b4d21e6f40ec7ece2d879c580acb2533b1" title="IMG_20230201_005625"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a2e32b4d21e6f40ec7ece2d879c580acb2533b1_2_690x213.jpeg" alt="IMG_20230201_005625" data-base62-sha1="8iGFFVB8KNRe1Ua2JynCFAK1FPr" width="690" height="213" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a2e32b4d21e6f40ec7ece2d879c580acb2533b1_2_690x213.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a2e32b4d21e6f40ec7ece2d879c580acb2533b1_2_1035x319.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a2e32b4d21e6f40ec7ece2d879c580acb2533b1_2_1380x426.jpeg 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20230201_005625</span><span class="informations">1898×586 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Khrystyna_Faryna: <p>I updated my version of pytorch to 1.10 and it started working.</p>