Uncoherent timing of convolution using CUDA events

remi-or-dtk: <p>Hello,<br>
I am trying to time a convolution operation through PyTorch, using either torch.convolution or torch.cudnn_convolution. In both cases, I am using cuda events to time the convolution operation. I have noticed that the frequency with which I refresh the convolution data (input and kernel) impacts the timing of the operation. This seems weird, as the only things between the CUDA events is the convolution itself. Would you have any idea of why this is?<br>
Below is the figures drawn to show this phenomenon (I have observed it, with various intensities, on various GPUs but I am limited to one image in this post) and the code used to draw it.</p>
<p>Windows, Torch 1.11.0, Cuda 11.3, RTX 3070 Laptop<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4957f8b845b283c0d8d414073f2e2a987aac7414.png" data-download-href="https://discuss.pytorch.org/uploads/default/4957f8b845b283c0d8d414073f2e2a987aac7414" title="RTX3070L"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4957f8b845b283c0d8d414073f2e2a987aac7414_2_690x459.png" alt="RTX3070L" data-base62-sha1="asPkG91r4zEqpCGMKaOpfABMJZa" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4957f8b845b283c0d8d414073f2e2a987aac7414_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4957f8b845b283c0d8d414073f2e2a987aac7414_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/9/4957f8b845b283c0d8d414073f2e2a987aac7414.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RTX3070L</span><span class="informations">1200×800 69.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">## Imports
from typing import *
import torch
from tqdm.auto import trange
import numpy as np
from torch import Tensor
from torch.cuda import Event
from torch.backends import cudnn
import matplotlib.pyplot as plt


## Constants for convolution / data
BATCH_SIZE = 32
OUT_CHANNELS = 16
FM_SIZE = 64
KERNEL_SIZE = 3

cudnn_convolution_kwargs = dict(
    padding=(1, 1), stride=(1, 1), dilation=(1, 1), groups=1, 
    benchmark=True, deterministic=True, allow_tf32=True
)


## Functions
def generate_data() -&gt; Tuple[Tensor, Tensor]:
    # Generate ImageNet-like input
    x = torch.normal(mean=0, std=1, size=(BATCH_SIZE, 3, 224, 224), device='cuda')
    x *= torch.tensor((0.229, 0.224, 0.225), device='cuda').reshape((1, 3, 1, 1))
    x += torch.tensor((0.485, 0.456, 0.406), device='cuda').reshape((1, 3, 1, 1))
    # Generate properly initialized convolution weight
    w = torch.zeros((OUT_CHANNELS, 3, KERNEL_SIZE, KERNEL_SIZE), device='cuda')
    torch.nn.init.xavier_normal_(w)
    return x, w

def time_convolution(iters: int,
                     op_per_iter: int,
                     data_refresh_rate: int,
                     warmup_duration: int, 
                     ) -&gt; List[float]:
    cudnn.benchmark = True
    times = []
    # Warmup
    x, w = generate_data()
    for i in range(warmup_duration):
        torch.cudnn_convolution(x, w, **cudnn_convolution_kwargs)
    # Timing loop
    for i in range(iters):
        # Refresh data if needed
        if not i % data_refresh_rate:
            x, w = generate_data()
        # Launch timing
        start, end = Event(True), Event(True)
        torch.cuda.synchronize()
        start.record()
        # Operation loop
        for j in range(op_per_iter):
            torch.cudnn_convolution(x, w, **cudnn_convolution_kwargs)
        # End timing
        end.record()
        end.synchronize()
        times.append(start.elapsed_time(end) / op_per_iter)
    return times


## Script
if __name__ == '__main__':

    # Runtime constants
    ITERS = 200
    WARMUP = 1000

    # Accumulators
    drrs: List[int] = []
    means: List[float] = []
    medians: List[float] = []

    # Data-gathering loop
    for drr in trange(1, ITERS+1):
        ts = time_convolution(iters=ITERS, op_per_iter=1, 
                            data_refresh_rate=drr, warmup_duration=WARMUP)
        drrs.append(drr)
        means.append(np.mean(ts))
        medians.append(np.median(ts))

    # Figure
    fig, ax = plt.subplots(1, 1, figsize=(12, 8))
    ax.plot(drrs, means, label='Mean')
    ax.plot(drrs, medians, label='Median')
    ax.set_title('Latency of convolution depending on its data refresh rate '
                 f"(measured on {ITERS} iterations)")
    ax.set_xlabel('Data refresh rate')
    ax.set_ylabel('Latency (ms)')
    ax.legend()
    plt.savefig('__tmp__.png')
    plt.show()
</code></pre>

remi-or-dtk: <p>Other figure, drawn with Linux and torch 1.13.1+cu117 and A100.</p>
<p>A100<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/319b1a8bb6e18c02ed2f711ab1905f084cb09d51.png" data-download-href="https://discuss.pytorch.org/uploads/default/319b1a8bb6e18c02ed2f711ab1905f084cb09d51" title="A100"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/319b1a8bb6e18c02ed2f711ab1905f084cb09d51_2_690x478.png" alt="A100" data-base62-sha1="74PHESUmFVXe1rp8wIDejP4rYn7" width="690" height="478" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/319b1a8bb6e18c02ed2f711ab1905f084cb09d51_2_690x478.png, https://discuss.pytorch.org/uploads/default/original/3X/3/1/319b1a8bb6e18c02ed2f711ab1905f084cb09d51.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/1/319b1a8bb6e18c02ed2f711ab1905f084cb09d51.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">A100</span><span class="informations">1010×701 44.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

remi-or-dtk: <p>Last figure, linux, torch 1.13.1+cu117, RTX3060<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc6a0a2ba0c1865459c6f63e66f855e88f0f7429.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc6a0a2ba0c1865459c6f63e66f855e88f0f7429" title="RTX_3060"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc6a0a2ba0c1865459c6f63e66f855e88f0f7429_2_690x478.png" alt="RTX_3060" data-base62-sha1="vrSepZDcjuaqIqaHw4h3PokvR7r" width="690" height="478" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc6a0a2ba0c1865459c6f63e66f855e88f0f7429_2_690x478.png, https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc6a0a2ba0c1865459c6f63e66f855e88f0f7429.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc6a0a2ba0c1865459c6f63e66f855e88f0f7429.png 2x" data-dominant-color="F9F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RTX_3060</span><span class="informations">1010×701 76.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

remi-or-dtk: <p>Hi, it seems this issue has avoided anyone’s attention!<br>
I have seen <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/richard">@richard</a> answer to similar question, would you be so kind to help?<br>
If anything’s missing from the issue, please let me know.<br>
Thanks.</p>

eqy: <p>I’m curious if this is actually due the data generation or something like power management ramping down clocks during “idle periods.” Could you check if e.g., replacing your data generation with something like <code>time.sleep(t)</code> where <code>t</code> is comparable to the time of data generation reproduces your results?</p>

remi-or-dtk: <p>I have tried to substitute the generation with a time.sleep calibrated on the warmup data generation (mean over 20 iterations) and these are the curves (GTX 2070):</p>
<p>Data re-generation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/6277dd5f248b24a304a7354d9acc1c7eecc4d08e.png" data-download-href="https://discuss.pytorch.org/uploads/default/6277dd5f248b24a304a7354d9acc1c7eecc4d08e" title="pytorch_issue_0_refresh_curve"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/6277dd5f248b24a304a7354d9acc1c7eecc4d08e_2_690x459.png" alt="pytorch_issue_0_refresh_curve" data-base62-sha1="e35BgFmLQOOC3dt86ouV03qckEK" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/6277dd5f248b24a304a7354d9acc1c7eecc4d08e_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/6277dd5f248b24a304a7354d9acc1c7eecc4d08e_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/2/6277dd5f248b24a304a7354d9acc1c7eecc4d08e.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_issue_0_refresh_curve</span><span class="informations">1200×800 65.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Sleep:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b2aeac6a690a7a412b874d5b45e663543b20777.png" data-download-href="https://discuss.pytorch.org/uploads/default/9b2aeac6a690a7a412b874d5b45e663543b20777" title="pytorch_issue_0_sleep_curve"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b2aeac6a690a7a412b874d5b45e663543b20777_2_690x459.png" alt="pytorch_issue_0_sleep_curve" data-base62-sha1="m8FXm7LTO9u7rg1bK3U6ds4nXWT" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b2aeac6a690a7a412b874d5b45e663543b20777_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b2aeac6a690a7a412b874d5b45e663543b20777_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b2aeac6a690a7a412b874d5b45e663543b20777.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_issue_0_sleep_curve</span><span class="informations">1200×800 38.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There is a similar effect. This would entail that the data generation is considered as a blocking idle period by the GPU, either because data is generated on CPU then moved to GPU or because the GPU is slow to generate random data. Would you have an idea why?<br>
Plus, the more idle periods → the faster the convolution goes is kind of illogical.<br>
Finaly, the behavior of the A100, for which the trend goes down instead of up, is also unexpected.</p>
<p>Does anyone have any idea to why this trend appears? Plus insights on the process of torch data generation?</p>

eqy: <p>On a laptop/mobile GPU setup, I would check if the GPU is thermally bottlenecked (and throttling down clock speeds), as more idle time would give the hardware a chance to cool down and ramp to higher clocks (at least initially).</p>