How "uneven" should the sequence lengths be for benefiting from nested tensors with the Transformer?

filipekstrm: <p>Hi,</p>
<p>I tried the <a href="https://docs.pytorch.org/tutorials/intermediate/transformer_building_blocks.html" rel="noopener nofollow ugc">tutorial</a> on accelerating the Pytorch Transformer with nested tensors. While I got the advertised speed-ups when running the code myself, I found that when using that type of implementation in my own project, the code was <strong>slower</strong> compared with regular nn.MultiHeadAttention and padded tensors. To try to verify that it was not just an implementation error on my part, I replaced the “sentence lengths” in the tutorial with a sample of sequence lengths from a batch of my own data, and I did indeed find that now using nested tensors was slower.</p>
<p>My question is therefore, firstly, is this expected (i.e., depending on how the distribution of sequence lengths looks like, it might be better to just go with padding)? Second, if this is indeed expected, is there a way (some heuristic, rule of thumb, theoretical work, etc) that informs me whether I should use nested tensors or padding?</p>

soulitzer: <p>I’m not sure if there are any results posted anywhere measuring perf when uneven-ness of seq lengths is varied, but there is a post here measuring general overhead of NT.<br>
TLDR is that NJT does bring some overhead by virtue of it being implemented in Python, but (1) can be ignored in compute-bound regime (large tensors) (2) can be compiled away with torch.compile.</p>
<aside class="onebox discoursetopic" data-onebox-src="https://dev-discuss.pytorch.org/t/initial-perf-results-from-compiling-nestedtensor-python-subclass-via-desugaring/1598">
  <header class="source">
      <img src="https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/optimized/1X/6cd7da56682d360e2c6006ff3e31eb250c5a8675_2_32x32.png" class="site-icon" alt="" width="32" height="32">

      <a href="https://dev-discuss.pytorch.org/t/initial-perf-results-from-compiling-nestedtensor-python-subclass-via-desugaring/1598" target="_blank" rel="noopener nofollow ugc" title="09:39PM - 23 October 2023">PyTorch Developer Mailing List – 23 Oct 23</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/352;"><img src="https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/optimized/2X/e/e808c76334c15df8ed0f35e06071da2a02bd2b5a_2_1024x523.png" class="thumbnail" width="690" height="352"></div>

<div class="title-wrapper">
  <h3><a href="https://dev-discuss.pytorch.org/t/initial-perf-results-from-compiling-nestedtensor-python-subclass-via-desugaring/1598" target="_blank" rel="noopener nofollow ugc">Initial perf results from compiling NestedTensor Python Subclass via desugaring</a></h3>
</div>

  <p>– with @ezyang, @bdhirsh, @jbschlosser, @cpuhrsch  Recently NestedTensor became compilable (stay tuned for a different post about this and our plans here), and just to get a sense of  what the numbers should look like, I did some measuring:  Intro...</p>

  <p>
    <span class="label1">Reading time: 2 mins 🕑</span>
      <span class="label2">Likes: 4 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


filipekstrm: <p>Hm, maybe I missunderstand what you mean by compiling it away, but in the code in the tutorial they do indeed use torch.compile and I still see this. Am I doing something wrong, or is it not meant literally that using compile will make the two options equal at worst?</p>