Inconsistent performance with large kernal convolutions between computers

aidanpcquinn: <p>Im having some troubles with large convolution kernals having extremly different performance between environments and computers.</p>
<p>Im using the following benchmark, which applies a large gaussian kernal two a large random array.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
import numpy as np
import time

def gaussian_kernel(size, sigma):

    size = int(size) // 2
    x, y = np.mgrid[-size:size+1, -size:size+1]
    g = np.exp(-(x**2 + y**2) / (2 * sigma**2))
    return g / g.sum()

def conv_helper(in_arr, conv_kernel, device, padding):
    start_time = time.time()

    # Convert input array to PyTorch tensor and move to device
    torch_in_arr = torch.from_numpy(in_arr).float().unsqueeze(0).unsqueeze(0)
    end_time = time.time()
    print(f"Time taken to convert input array to tensor: {end_time - start_time:.6f} seconds")

    start_time = time.time()
    torch_in_arr = torch_in_arr.to(device)
    end_time = time.time()
    print(f"Time taken to move input tensor to device: {end_time - start_time:.6f} seconds")

    start_time = time.time()
    # Convert kernel to PyTorch tensor and move to device
    torch_kernel = torch.from_numpy(np.array(conv_kernel)).float().unsqueeze(0).unsqueeze(0)
    end_time = time.time()
    print(f"Time taken to convert kernel to tensor: {end_time - start_time:.6f} seconds")

    start_time = time.time()
    torch_kernel = torch_kernel.to(device)
    end_time = time.time()
    print(f"Time taken to move kernel to device: {end_time - start_time:.6f} seconds")

    torch.cuda.synchronize() 
    start_time = time.time()
    # Perform convolution on the GPU
    out_arr_gpu = F.conv2d(torch_in_arr, torch_kernel, padding=padding)
    torch.cuda.synchronize() 
    end_time = time.time()
    print(f"Time taken for convolution operation: {end_time - start_time:.6f} seconds")

    torch.cuda.synchronize() 
    start_time = time.time()
    # Move result back to CPU asynchronously
    out_arr_gpu = out_arr_gpu.to('cpu', non_blocking=True)
    torch.cuda.synchronize()  # Ensure all GPU operations are completed
    end_time = time.time()
    print(f"Time taken to move result back to CPU: {end_time - start_time:.6f} seconds")

    start_time = time.time()
    # Convert to numpy array
    out_arr_cpu = out_arr_gpu.numpy().squeeze()
    end_time = time.time()
    print(f"Time taken to convert to numpy array: {end_time - start_time:.6f} seconds")

    return out_arr_cpu

# Example usage
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
input_size = 4000
kernel_size = 151
sigma = 25

# Create a large random input array
input_array = np.random.randn(input_size, input_size)

# Create a Gaussian kernel
gaussian_k = gaussian_kernel(kernel_size, sigma)

# Perform convolution with different paddings
padding_same = kernel_size // 2  # Same padding

print("Running with same padding:")
output_array_same = conv_helper(input_array, gaussian_k, device, padding_same)
</code></pre>
<p>I have tested this in multiple enironments:</p>
<ol>
<li>PC1 1x3090, pytorch 2.1.2, Cuda 12.2, Driver 535…</li>
<li>PC1 but in a NVCR pytorch container 22.05 (torch 1.12)</li>
<li>PC2 2x4090, pytorch 2.1.2, Cuda 12.2, Driver 535…</li>
<li>PC2 but in a NVCR pytorch container 22.05 (torch 1.12)</li>
</ol>
<p>1,2,4 all get a total runtime of 2 ish seconds with the convolution operation taking less than 1 second. But specifically 3 has the convolution operation take 20 seconds.</p>
<p>Any further ways i could narrow down why setup 3 is so much slower?</p>
<p>I have also tried different pytorch environments, manually setting a gpu device to a numa node (both setups are AMD threadrippers, 3960x and 7960x respectivly), disabling NCCL_P2P…</p>
<p>It seems like it must have to be some cuda or torch system variable, as the performance is as expected in the docker container</p>
<p>Cheers.</p>

aidanpcquinn: <p>More testing.</p>
<p>PC1 has a conv time of .5 with nvcr pytorch container 24.01,<br>
(CUDA 12.3.2, torch 2.2.0a0+81ea7a4, TensorRT 8.6.1.6)</p>
<p>But PC2 has the same 20 second time with that same container</p>
<p>Furthermore,<br>
When using python versions &lt; 2 (tested multiple containers on both pcs), they have the same performance.</p>

ptrblck: <p>You could profile the workload via e.g. Nsight Systems to narrow down which operations are causing the slowdown. Assuming you are right and it’s a conv, you could check the cuDNN version and update it to the latest one.</p>

aidanpcquinn: <p>I will try both thanks,</p>
<p>Can cudnn version on the host effect torch within a container?</p>

ptrblck: <p>If you mount it to the container to replace the ones shipped in the container and could thus use the host version. By default, docker should not be able to use your host cuDNN.</p>

aidanpcquinn: <p>Here are two screen shots from Nsight</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/395cbaf73672a44060503dc1dc0b9a09aff59ef5.png" data-download-href="https://discuss.pytorch.org/uploads/default/395cbaf73672a44060503dc1dc0b9a09aff59ef5" title="Screenshot from 2024-06-08 00-12-16"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/395cbaf73672a44060503dc1dc0b9a09aff59ef5_2_690x456.png" alt="Screenshot from 2024-06-08 00-12-16" data-base62-sha1="8brT9MrqtDs59grUmYeMQ40SR1j" width="690" height="456" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/395cbaf73672a44060503dc1dc0b9a09aff59ef5_2_690x456.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/395cbaf73672a44060503dc1dc0b9a09aff59ef5_2_1035x684.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/395cbaf73672a44060503dc1dc0b9a09aff59ef5_2_1380x912.png 2x" data-dominant-color="C6E5ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-06-08 00-12-16</span><span class="informations">1797×1189 307 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/142b702427e127dd927b1bf1465ce32317033230.png" data-download-href="https://discuss.pytorch.org/uploads/default/142b702427e127dd927b1bf1465ce32317033230" title="Screenshot from 2024-06-08 00-12-00"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/142b702427e127dd927b1bf1465ce32317033230_2_690x456.png" alt="Screenshot from 2024-06-08 00-12-00" data-base62-sha1="2SqC852cYtgYNZmZJDYIP7i0zvy" width="690" height="456" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/142b702427e127dd927b1bf1465ce32317033230_2_690x456.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/142b702427e127dd927b1bf1465ce32317033230_2_1035x684.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/142b702427e127dd927b1bf1465ce32317033230_2_1380x912.png 2x" data-dominant-color="C3EFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-06-08 00-12-00</span><span class="informations">1797×1189 262 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

aidanpcquinn: <p>These are from PC1 that is performing way better</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/198c216d3da925a1dc9192ce7ef5d9cb49c3824d.png" data-download-href="https://discuss.pytorch.org/uploads/default/198c216d3da925a1dc9192ce7ef5d9cb49c3824d" title="Screenshot from 2024-06-08 00-28-38"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/198c216d3da925a1dc9192ce7ef5d9cb49c3824d_2_690x459.png" alt="Screenshot from 2024-06-08 00-28-38" data-base62-sha1="3E0aeKZmhQScmHaLUGhfuou8jCJ" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/198c216d3da925a1dc9192ce7ef5d9cb49c3824d_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/198c216d3da925a1dc9192ce7ef5d9cb49c3824d_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/198c216d3da925a1dc9192ce7ef5d9cb49c3824d_2_1380x918.png 2x" data-dominant-color="ECF0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-06-08 00-28-38</span><span class="informations">1611×1073 154 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/571452f414af705a3fd1f28fd8f144605c69f538.png" data-download-href="https://discuss.pytorch.org/uploads/default/571452f414af705a3fd1f28fd8f144605c69f538" title="Screenshot from 2024-06-08 00-29-33"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/571452f414af705a3fd1f28fd8f144605c69f538_2_690x467.png" alt="Screenshot from 2024-06-08 00-29-33" data-base62-sha1="cql5cO884eo4WChxaemOUnROtEQ" width="690" height="467" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/571452f414af705a3fd1f28fd8f144605c69f538_2_690x467.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/571452f414af705a3fd1f28fd8f144605c69f538_2_1035x700.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/571452f414af705a3fd1f28fd8f144605c69f538_2_1380x934.png 2x" data-dominant-color="DDEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-06-08 00-29-33</span><span class="informations">1623×1100 180 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

aidanpcquinn: <p>Additionally, reinstalling cuDNN made no difference.</p>

aidanpcquinn: <p>Getting the same behaviour on a fresh install of ubuntu 22.04, but i think i may have solved whats going on.</p>
<p>So interestingly looking at nSight systems and running the same conv operation 3x with torch.backends.cudnn.benchmark=True. The method that its picking on the first system and after optimizing with benchmark is ‘conv2d_grouped_direct_kernel’, while PC2 by default selects an operation that is much slower for this kernal/matrix size. Question now is, is there a way to force the use of the different method under the hood?</p>

aidanpcquinn: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/1/71f61537d27d8d8ff83181ece81547b78e35e49f.png" data-download-href="https://discuss.pytorch.org/uploads/default/71f61537d27d8d8ff83181ece81547b78e35e49f" title="Screenshot from 2024-06-08 20-52-17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71f61537d27d8d8ff83181ece81547b78e35e49f_2_690x420.png" alt="Screenshot from 2024-06-08 20-52-17" data-base62-sha1="gg9bqtiu37fbPpZtqaDa6NEFgIv" width="690" height="420" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71f61537d27d8d8ff83181ece81547b78e35e49f_2_690x420.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71f61537d27d8d8ff83181ece81547b78e35e49f_2_1035x630.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71f61537d27d8d8ff83181ece81547b78e35e49f_2_1380x840.png 2x" data-dominant-color="E3E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-06-08 20-52-17</span><span class="informations">1794×1093 301 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f11f6a678f3d2c658475e8ee34c926bcd7443be.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f11f6a678f3d2c658475e8ee34c926bcd7443be" title="Screenshot from 2024-06-08 20-51-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f11f6a678f3d2c658475e8ee34c926bcd7443be_2_690x420.png" alt="Screenshot from 2024-06-08 20-51-58" data-base62-sha1="kpELq4BqQ8pssauChKt6di305mC" width="690" height="420" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f11f6a678f3d2c658475e8ee34c926bcd7443be_2_690x420.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f11f6a678f3d2c658475e8ee34c926bcd7443be_2_1035x630.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f11f6a678f3d2c658475e8ee34c926bcd7443be_2_1380x840.png 2x" data-dominant-color="DFE5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-06-08 20-51-58</span><span class="informations">1794×1093 274 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>For static shapes you could use <code>torch.backends.cudnn.benchmark = True</code> to allow cuDNN to profile kernels before selecting them. There is no way to manually select algorithms.</p>