Tracing: Confusion with duration of similar `flip` operations

username: eduardo4jesus
cooked: <p>I was analyzing a tracing from one of my experiments. At a given point I have two consecutive <code>aten::flip</code> operations with the respective input dimensions:</p>
<ol>
<li><code>[[128, 128, 64, 6], []]</code></li>
<li><code>[[128, 128, 64, 13, 6], []]</code></li>
</ol>
<p>This would be the code associated with it</p>
<pre><code class="lang-auto">    Y = torch.nn.functional.pad(Y, padding)
    if Y.ndim == 5:
      Y[:, :, :,  0, m:] = _Y[:, :, :, 0, 1:1+p].flip([-1])
      Y[:, :, :, 1:, m:] = _Y[:, :, :, 1:, 1:1+p].flip([-2, -1]);
    elif Y.ndim == 4:
      Y[:, :,  0, m:] = _Y[ :, :, 0, 1:1+p].flip([-1])
      Y[:, :, 1:, m:] = _Y[ :, :, 1:, 1:1+p].flip([-2, -1]);
    else:
      raise ValueError('Wrong Shape for Y')
</code></pre>
<p>For some reason the first takes more time to execute than the second. This is counter-intuitive for me given the first has less elements. <strong>Why would this be?</strong></p>
<p>Here are some screen shots from the tracing.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b33a6e762782ab3ec78a7d48ddbf8bb9a731b08a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b33a6e762782ab3ec78a7d48ddbf8bb9a731b08a" title="Screen Shot 2021-06-25 at 5.58.07 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33a6e762782ab3ec78a7d48ddbf8bb9a731b08a_2_690x396.jpeg" alt="Screen Shot 2021-06-25 at 5.58.07 AM" data-base62-sha1="pzwEG3huEW3aW3FXS48flEfhogy" width="690" height="396" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33a6e762782ab3ec78a7d48ddbf8bb9a731b08a_2_690x396.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33a6e762782ab3ec78a7d48ddbf8bb9a731b08a_2_1035x594.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33a6e762782ab3ec78a7d48ddbf8bb9a731b08a_2_1380x792.jpeg 2x" data-dominant-color="EAE9EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-06-25 at 5.58.07 AM</span><span class="informations">2032×1169 278 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba6b639c23d47a271f3f4d369de708864a92839f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ba6b639c23d47a271f3f4d369de708864a92839f" title="Screen Shot 2021-06-25 at 5.58.09 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b639c23d47a271f3f4d369de708864a92839f_2_690x396.jpeg" alt="Screen Shot 2021-06-25 at 5.58.09 AM" data-base62-sha1="qB8T9AE3odhVFZXf8dcemwIpv2L" width="690" height="396" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b639c23d47a271f3f4d369de708864a92839f_2_690x396.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b639c23d47a271f3f4d369de708864a92839f_2_1035x594.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b639c23d47a271f3f4d369de708864a92839f_2_1380x792.jpeg 2x" data-dominant-color="EBEAEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-06-25 at 5.58.09 AM</span><span class="informations">2032×1169 278 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am pretty new to tracing and understanding all the dynamics of GPU kernel dispatches and optimization. Is there anything I am missing that would make things more clear here?</p>

username: ptrblck
cooked: <p>I cannot reproduce the results using this benchmark code:</p>
<pre><code class="lang-python">def fun1(Y, p=3, m=-3):
    Y[:, :, :,  0, m:] = Y[:, :, :, 0, 1:1+p].flip([-1])
    Y[:, :, :, 1:, m:] = Y[:, :, :, 1:, 1:1+p].flip([-2, -1])
    return Y


def fun2(Y, p=3, m=-3):
    Y[:, :,  0, m:] = Y[ :, :, 0, 1:1+p].flip([-1])
    Y[:, :, 1:, m:] = Y[ :, :, 1:, 1:1+p].flip([-2, -1])
    return Y

Y1 = torch.randn([128, 128, 64, 13, 6], device='cuda')
Y2 = torch.randn([128, 128, 64, 6], device='cuda')

t1 = torch.utils.benchmark.Timer(stmt="fun1(Y1)", globals=globals())
t2 = torch.utils.benchmark.Timer(stmt="fun2(Y2)", globals=globals())

t1.blocked_autorange()
&gt; &lt;torch.utils.benchmark.utils.common.Measurement object at 0x7fe04040ad30&gt;
  fun1(Y1)
    2.28 ms
    1 measurement, 100 runs , 1 thread
t2.blocked_autorange()
&gt; &lt;torch.utils.benchmark.utils.common.Measurement object at 0x7fe04040a070&gt;
  fun2(Y2)
    Median: 226.38 us
    IQR:    6.76 us (224.14 to 230.90)
    8 measurements, 100 runs per measurement, 1 thread
</code></pre>
<p>Could you check, if both tensors are contiguous before passed to the functions?</p>

username: eduardo4jesus
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>, I really appreciate that you are taking a look into this. Thank you so much.</p>
<p>So, here is a code snippet for reproducing it.</p>
<pre><code class="lang-auto">import torch

def my_not_so_secrete_function(Y, _Y):
  p = Y.shape[-2] - Y.shape[-1]
  padding = (0, p)
  m = Y.shape[-1]

  Y = torch.nn.functional.pad(Y, padding)
  if Y.ndim == 5:
    Y[:, :, :,  0, m:] = _Y[:, :, :, 0, 1:1+p].flip([-1])
    Y[:, :, :, 1:, m:] = _Y[:, :, :, 1:, 1:1+p].flip([-2, -1]);
  elif Y.ndim == 4:
    Y[:, :,  0, m:] = _Y[ :, :, 0, 1:1+p].flip([-1])
    Y[:, :, 1:, m:] = _Y[ :, :, 1:, 1:1+p].flip([-2, -1]);
  else:
    raise ValueError('Wrong Shape for Y')

  return Y

device = torch.device('cuda' if torch.cuda.is_available else 'cpu')
with torch.profiler.profile(
   record_shapes=True
) as profile:
  for _ in range(8):
    Y = torch.randint(32, size=(128, 64, 3, 32, 17), dtype=torch.float, device=device)
    _Y = torch.randint(32, size=(128, 64, 3, 32, 17), dtype=torch.float, device=device)
    output = my_not_so_secrete_function(Y, _Y)

profile.export_chrome_trace('profile-investigation.json')
</code></pre>
<p>I uploaded the <code>profile-investigation.json</code> file <a href="https://gist.github.com/eduardo4jesus/d8c022d452fdc0e196a55104c6b50a6d" rel="noopener nofollow ugc">here as a Github Gist.</a></p>