Conv3d tensor core utilisation

username: ChSonnabend
cooked: <p>Hi,</p>
<p>I have a problem concerning the inference of models with Conv3d layers on Nvidia (V100) and AMD (MI100) GPU’s. It seems like the tensor cores are not utilised at all by this layer. I tried again with a fully connected layer and could verify the utilisation using nvprof. Below, I try to provide a minimal code example (speed.py)</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn

class test_network(nn.Module):

    def __init__(self, window_size=11):
        
        super().__init__()
        self.fc1 = nn.Linear(7*7*7, 128)
        self.relu = nn.ReLU()
    
    def forward(self, X):
        return self.relu(self.conv11(X))

class test_network(nn.Module):

    def __init__(self):
        
        super().__init__()
        self.conv1 = nn.Conv3d(in_channels=1, out_channels=8, kernel_size=(3,3,3), padding=(1,0,0))
        self.relu = nn.ReLU()
    
    def forward(self, X):
        return self.relu(self.fc1(X))

test_net = test_network()

number_elements = 1000000
number_loops = 50

dtype=torch.float16
device=("cuda" if torch.cuda.is_available() else "cpu")

print("Cuda is available:", torch.cuda.is_available())

test_net = test_net.to(dtype=dtype, device=device)


with torch.no_grad():
    for i in range(number_loops):
        torch.cuda.synchronize()
        test = torch.ones((number_elements,1,7,7,7), dtype=dtype, device=device)
        out = test_net(test)
        torch.cuda.synchronize()
        del test
</code></pre>
<p>For using this script, just (un-)comment the model that is to be tested and adapt the input size (either (number_elements,1,7,7,7) for the convolutional network or (number_elements,7 * 7 * 7) for the fully connected network).</p>
<p>I then test the utilisation (Nvidia machine) using<br>
<code>sudo -E /usr/local/cuda/bin/nvprof -m tensor_precision_fu_utilization /usr/bin/python3 speed.py</code></p>
<p>My setup: Nvidia V100 PCle, Cuda Driver version 12.2, latest PyTorch (pip) installation.</p>
<p>Below you can find a screenshot of nvprof for the Conv3d model. (Unfortunately, as a new user, I cannot embed more than one image in my post).</p>
<p>(Conv model)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/3508f854553158edd898caf4a46836f3d26563b1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3508f854553158edd898caf4a46836f3d26563b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3508f854553158edd898caf4a46836f3d26563b1_2_690x89.jpeg" alt="image" data-base62-sha1="7zawuPooXVPCBRcbE56Iq87Hwvn" width="690" height="89" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3508f854553158edd898caf4a46836f3d26563b1_2_690x89.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3508f854553158edd898caf4a46836f3d26563b1_2_1035x133.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3508f854553158edd898caf4a46836f3d26563b1_2_1380x178.jpeg 2x" data-dominant-color="143850"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×248 65 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have not explicitly tested something like nvprof on the AMD machines, however I see similar numbers in terms of TFLOPs, so I assume the problem originates from the same source.</p>
<p>Any help is much appreciated.</p>
<p>Many thanks, kind regards,<br>
Christian</p>