Why `torch.histc` is slow on first sample?

username: sbelharbi
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.histc.html" rel="noopener nofollow ugc">torch.histc</a> performed over a minibatch  cuda tensor of n samples is slow on first sample…<br>
is that normal?</p>
<p><code>torch.histc</code> takes way too long on the first sample, .34sec<br>
and 0.000163sec on the left.</p>
<p>any idea why? and how to speedup this?<br>
the code is in the middle of large code where warmup has already been done…<br>
thanks</p>

username: ptrblck
cooked: <p>Could you describe how you’ve profiled the code, please?<br>
Often profiling code misses e.g. synchronizations and could thus accumulate previously launched kernel times into the current timer and would thus return wrong timings.</p>

username: sbelharbi
cooked: <p>i re-run the timing with cuda.event, and i get relatively the same time except the first sample when using python time i get way higher time than torch.cuda.event. i dont know why is that.<br>
not sure what python time was computing? and why this huge time at the first sample? this happens constantly on every first sample of any minibatch.</p>
<p>what is the true time that my code will take (it cant be 2 different times)? is it the one computed with torch.cuda.event or python time? if python time is taking this long, someone must be doing something that torch.cuda.event didnt take in consideration… it could be something to do with synchronization. when serializing the execution, i get consistent time in both timers.</p>
<p>could torch.histc creates a synchronization point at the first sample allowing all the kernels over other samples to be done, therefore it is fast on the rest of samples but not the first one?</p>
<p>the goal of this measurements is to find bottlenecks to improve their speed and speedup the full code.<br>
in this case, for every minibatch, 99% of time is spent on first sample everytime. it is fine if this happens only once. but this happens in every first sample of every minibatch. so, it costs.</p>
<p>thanks</p>
<p>here is all the timings:</p>
<ul>
<li>time using time.perf_counter  (s)</li>
</ul>
<ol>
<li>with CUDA_LAUNCH_BLOCKING=1</li>
</ol>
<pre><code class="lang-auto">sample 1 histc 0.0004032440483570099 
sample 2 histc 0.0005275830626487732  
sample 3 histc 0.000552598387002945 
</code></pre>
<ol start="2">
<li>without  CUDA_LAUNCH_BLOCKING=1</li>
</ol>
<pre><code class="lang-auto">sample 1 histc 0.3368970938026905  **WHY?** 
sample 2 histc 0.00043926388025283813 
sample 3 histc 0.0003735348582267761 
</code></pre>
<ul>
<li>time using torch.cuda.Event (ms)</li>
</ul>
<ol>
<li>with CUDA_LAUNCH_BLOCKING=1</li>
</ol>
<pre><code class="lang-auto">sample 1 histc 0.4039680063724518
sample 2 histc 0.6740800142288208
sample 3 histc 5.139200210571289
</code></pre>
<ol start="2">
<li>without  CUDA_LAUNCH_BLOCKING=1</li>
</ol>
<pre><code class="lang-auto">sample 1 histc 0.3036479949951172
sample 2 histc 0.47996801137924194
sample 3 histc 0.5324159860610962
</code></pre>
<p>general setup:</p>
<pre><code class="lang-python">def function(tensor):
    # operations ...
    # tx = time.perf_counter()
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    start_event.record()

    hist = torch.histc(x, bins=bins)
    
    end_event.record()
    elapsed_time_ms = start_event.elapsed_time(end_event)
    print('histc {}'.format(elapsed_time_ms))
    # print('histc {} '.format(time.perf_counter() - tx))
  
for epoch in epochs:
  forward(minibatch)
  # prepare tensors
  l_tensors = None
  for t in range(minibatch_size):
    l_tensors.vstack((l_tesnors, compute(tensor_t))
   
  # process all tensors.
  for tensor in l_tensors:
    function(tensor) # &lt;-- where we measure time.
</code></pre>

username: gcramer23
cooked: <p>Hi the pytorch profiler might be beneficial to you <a href="https://pytorch.org/docs/stable/profiler.html" class="inline-onebox" rel="noopener nofollow ugc">torch.profiler — PyTorch 1.9.0 documentation</a>. Also you can use the trace functionality to identify bottlenecks in your code <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html#using-tracing-functionality" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 1.9.0+cu102 documentation</a>.</p>

username: sbelharbi
cooked: <p>i think i arrived to timing the histc function specifically and its parent function where it was called  through the pytorch profiler.<br>
it doesnt detail all the functions i guess. so, i had to manually further time some functions.<br>
the profiler also showed that min and max tensor take the cake in time. but i think it is because i used them in the evaluation of the condition in <code>if</code> which is blocking cause of gpu-2-cpu transfer.</p>

username: ptrblck
cooked: <p>Based on your description you might profile the CUDA init in the first iteration using the Python timer or the “wake up cycle” in case the GPU was idle before, so I would recommend to use <code>torch.utils.benchmark</code> to profile specific ops, as it’ll take care of the warmup iterations as well as necessary synchronizations.</p>

username: sbelharbi
cooked: <p>in all the results above, <code>torch.utils.benchmark</code> is set to true.</p>
<p>the slowness over the first sample of minbatch is recurrent over all minibatchs not just the first one.<br>
so, the warmup couldnt be the issue.</p>
<p>i did some more digging, and now i am sure that the slowness is caused by cuda implicit synchronization. it is not torch.histc, tensor.min/max, or python <code>if</code>.</p>
<p>to clarify, i work on a tensor <code>z</code> of size (32, 1, h, w) that is loaded from disk using pytorch dataloader sample by sample with multiple workers and stitched together the same way we do with loading images to form a minibatch.</p>
<p>after loading our tensor <code>z</code>, i do the only following cuda operations on it very early in the code:</p>
<pre><code class="lang-python"># z is loaded using datalaoder
with torch.no_grad():
  # z: (bsz, 1, h, w)
  assert z.ndim == 4
 # Quick fix
 z = torch.nan_to_num(z, nan=0.0, posinf=1., neginf=0.0)
 z = F.interpolate(z,
                   image_size,
                   mode='bilinear',
                   align_corners=False)  # (bsz, 1, h, w)
</code></pre>
<p>and that is all.</p>
<p>after this, i do some other stuff that do not involve <code>z</code> at all, such as forward in model.<br>
then, i call a function that iterate on each sample in <code>z</code> and do some cuda stuff.<br>
what i learned is that the FIRST CUDA OPERATION ON THE FIRST SAMPLE will cause A CUDA IMPLICIT SYNCHRONIZATION. either it is z.min()/max, torch.histc, python if min=max, … it does not matter. the moment cuda is involved, it calls synch. it is like kernels on <code>z</code> are not done!!!</p>
<pre><code class="lang-python"># prepare z
# other stuff that do not involve z...

def func(z_):
  for i in range(bsz):
     process(z_[i])

func(z)
</code></pre>
<p>ok, let say they are not done.<br>
if and only if add <code>torch.cuda.synchronize()</code> only right before starting the loop, all samples will run in .0003sec otherwise, the first will run in .33se while the rest in .0003sec. meaning the synch works, and <code>z</code> is ready.<br>
but if ask for synch right after preparing <code>z</code>, it wont work!!!<br>
later when staring working on <code>z</code>, cuda will synch on its own… it is so weird.</p>
<pre><code class="lang-python"># prepare z
# torch.cuda.synchronize()  # does not work here. we get the same slow behavior.
# other stuff that do not involve z...

def func(z_):
  torch.cuda.synchronize() # will work here. z will be ready and no need for cuda implicit synch.
  for i in range(bsz):
     process(z_[i])

func(z)
</code></pre>
<p>this implicit cuda synch happens even if i create z as zero right before the loop such</p>
<pre><code class="lang-python"># prepare z

# other stuff that do not involve z...

def func(z_): 
  # forget about our z... lets create one new one full of 0.
  z_ = torch.zeros((bsz, 1, h, w), dtype=torch.float, device=cuda0)
  for i in range(bsz):
     process(z_[i])

func(z)
</code></pre>
<p>the same thing will happen, first sample slow, then the rest is fast.<br>
in the loop, i the only manipulated cuda tensor  z. no external tensors are required.<br>
i dont understand what’s blocking cuda, and why it needs to synch.<br>
<code>z</code> should be ready by then, and even creating zero tensor should be instant.</p>
<p>– creation of z and func are wrapped within torch.no_grad(). i tried with and without, i get the same pattern. here, i simplified <code>func</code>, it is actually a class <code>torch.nn.Module</code>, and i call forward.</p>
<p>any advice<br>
thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sbelharbi" data-post="7" data-topic="130283">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/48.png" class="avatar"> sbelharbi:</div>
<blockquote>
<p>in all the results above, <code>torch.utils.benchmark</code> is set to true.</p>
</blockquote>
</aside>
<p><code>torch.utils.benchmark</code> is a utility function used to profile operations and is not a flag, so setting it to <code>True</code> won’t do anything besides overriding it.</p>
<p>Based on your description, you are getting the right profiling by synchronizing before starting and stopping the timers and get the expected wrong results otherwise.</p>

username: sbelharbi
cooked: <p>did more digging.<br>
didnt arrive to the exact cause of the problem, until now.</p>
<p>note, i have a network where i perform a forward for training over minibatch <code>x</code>, then i have the part of the code that processes <code>z</code>.</p>
<p>the network forward operates directly on minibatch  (parallel).<br>
however, the part operating on <code>z</code>, works sequentially, only at the end, i gathered what batchable into single cuda operation over entire batch.</p>
<p>initially, i perform:</p>
<pre><code class="lang-python">network(x)
process(z)
</code></pre>
<p>this causes the discussed issue when arriving to <code>process(z)</code> where first sample is very slow .33sec, then rest of samples are fast .003 with mintach times to .45sec.</p>
<p>then, i switched the order to</p>
<pre><code class="lang-python">process(z)
network(x)
</code></pre>
<p>things works properly. problem ‘disappeared’ at <code>process(z)</code>. all samples in <code>z</code> are processed in .003 and total <code>z</code> in .11sec without an explicit <code>torch.cuda.synchronize()</code>.</p>
<p>but that’s not all.</p>
<p><code>network(x)</code> is completely independent from <code>process(z)</code> in term of data or functions. so, i dont know how the order of executing kernels will affect them. also, this doesnt necessarily mean the problem is solved because the overall time of 1 epoch didnt improve much (from 2min14s to 2min03s, i expected more gain, like .33se * 188 minibatches). cuda is probably doing some implicit synch.</p>
<p>so, did some more digging. it turns out i was wrong about the time of <code>network(x)</code>. it is not .001, but .33sec. i used the wrong tool to measure time cuda stuff using python time.</p>
<p>so, the extra .33sec were coming from the network forward and not other operations.<br>
the order does not matter.<br>
cuda needs to synch to get the model output causing the delay.</p>
<p>sorry for the inconvenience, and thanks for your help.</p>

username: sbelharbi
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="130283">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.utils.benchmark</code> is a utility function used to profile operations and is not a flag, so setting it to <code>True</code> won’t do anything besides overriding it.</p>
</blockquote>
</aside>
<p>sorry, i was thinking of <code>torch.backends.cudnn.benchmark</code> which is flag.</p>