Performance regression in advanced indexing

jens: <p>Hello people, thanks for the amazing library. I am working on a simulation package which uses pytorch mostly as a numpy replacement, so not an ML use case. We found performance regressions in the advanced indexing going from 1.10.1 to 2.0.1.The torch versions are pip installed. We tested x86_64 and arm.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2ee1b42c06c05022f524261c535dc71674297f68.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ee1b42c06c05022f524261c535dc71674297f68" title="x86_64_py3.8.16_read_indices"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2ee1b42c06c05022f524261c535dc71674297f68_2_690x394.png" alt="x86_64_py3.8.16_read_indices" data-base62-sha1="6GJwYdER1ggHrUpd9WrOC8aAw3K" width="690" height="394" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2ee1b42c06c05022f524261c535dc71674297f68_2_690x394.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2ee1b42c06c05022f524261c535dc71674297f68_2_1035x591.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2ee1b42c06c05022f524261c535dc71674297f68.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">x86_64_py3.8.16_read_indices</span><span class="informations">1226×701 54.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The python benchmark is attached. Each test was run more than 10k times (after a warmup of 10k iterations) and the minimum from each run was selected. All tests are basically indexing operations into arrays for reading or writing. In comparison we also tested the same operations using numpy. Tensor size is (1000,) and a random bitmask with 500 true entries was generated, which was used for all calculations.</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/JensWehner/3524a121f42a3ec912fa8f843249b75d">
  <header class="source">

      <a href="https://gist.github.com/JensWehner/3524a121f42a3ec912fa8f843249b75d" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/JensWehner/3524a121f42a3ec912fa8f843249b75d" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/JensWehner/3524a121f42a3ec912fa8f843249b75d</a></h4>

  <h5>tensor_tests.py</h5>
  <pre><code class="Python"># type: ignore
"""Performance tests for tensor access variations."""
from typing import Callable, List, Union

import itertools
import numpy as np
import pytest
import torch

</code></pre>
   This file has been truncated. <a href="https://gist.github.com/JensWehner/3524a121f42a3ec912fa8f843249b75d" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">pytest --randomly-seed=1 --benchmark-verbose --benchmark-only --benchmark-disable-gc --benchmark-warmup=on --benchmark-warmup-iterations=10000 -m torchperformance test_performance_tensor_access.py
</code></pre>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="jens" data-post="1" data-topic="182445">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/d2c977/48.png" class="avatar"> jens:</div>
<blockquote>
<p>Each test was run more than 10k times and the minimum from each run was selected.</p>
</blockquote>
</aside>
<p>How does the mean look after performing proper warmup iterations?</p>

jens: <p>So using the average instead of the min we got this result. The warmup was 10k. Do you have any experience what kind of warm-up is sufficient?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbfd7b5964765789f038906b3ece06e6332606ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/fbfd7b5964765789f038906b3ece06e6332606ac" title="x86_64_py3.8.16_read_indices"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbfd7b5964765789f038906b3ece06e6332606ac_2_690x394.png" alt="x86_64_py3.8.16_read_indices" data-base62-sha1="zXcVuWwfrr3iYkhvlWMWfVgjVEg" width="690" height="394" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbfd7b5964765789f038906b3ece06e6332606ac_2_690x394.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbfd7b5964765789f038906b3ece06e6332606ac_2_1035x591.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbfd7b5964765789f038906b3ece06e6332606ac.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">x86_64_py3.8.16_read_indices</span><span class="informations">1226×701 59.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>