Why libtorch Tensor indexing much slower than C++ vector?

SeanZhang777: <h2>I find that indexing an at::Tensor using libtorch is much slower than indexing a C++ vector, is that normal?</h2>
<h2>Demo code like:</h2>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;string&gt;
#include &lt;chrono&gt;

int main(int argc, const char* argv[]) {
  at::Tensor test_tensor(at::zeros({16000, 100, 4}, at::kFloat));
  auto start = std::chrono::steady_clock::now();
  for (int i = 0; i &lt; 80000; ++i) {
    auto idx = test_tensor[15999][0][0];
  }
  auto end = std::chrono::steady_clock::now();
  double time = std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(end - start).count();
  std::cout &lt;&lt; "tensor indexing cost " &lt;&lt; time &lt;&lt; " ms." &lt;&lt; std::endl;

  std::vector&lt;float&gt; v1(4);
  std::vector&lt;std::vector&lt;float&gt;&gt; v2(100, v1);
  std::vector&lt;std::vector&lt;std::vector&lt;float&gt;&gt;&gt; test_vector(16000, v2);
  start = std::chrono::steady_clock::now();
  for (int i = 0; i &lt; 80000; ++i) {
    auto idx = test_vector[15999][0][0];
  }
  end = std::chrono::steady_clock::now();
  time = std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(end - start).count();
  std::cout &lt;&lt; "vector indexing cost " &lt;&lt; time &lt;&lt; " ms." &lt;&lt; std::endl;
}
</code></pre>
<h2>ouput is:</h2>
<pre><code class="lang-auto">tensor indexing cost 153 ms.
vector indexing cost 0 ms.
</code></pre>
<h2>Environment</h2>
<p>ubuntu 18.04<br>
cmake 3.15.5<br>
g++ 7.5.0<br>
libtorch 1.6.0+cu101<br>
GPU Intel® Core™ i7-8750H</p>

Lin_Jia: <p>See this:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/45139" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/45139" target="_blank" rel="nofollow noopener">Libtorch index API perform poorly under repetitive usage</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-09-22" data-time="17:04:32" data-timezone="UTC">05:04PM - 22 Sep 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/linjiapro" target="_blank" rel="nofollow noopener">
          <img alt="linjiapro" src="https://avatars0.githubusercontent.com/u/4970879?v=4" class="onebox-avatar-inline" width="20" height="20">
          linjiapro
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
I have a Libtorch project that has some execution speed issues. I have spent half a day to debug it....</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Accessor is much faster but still slower than vector. Regarding why index is so slow, look at this:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/1494005cfdf228f8441c1a7aecdfe1c3dfc77de6/aten/src/ATen/TensorIndexing.cpp" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/1494005cfdf228f8441c1a7aecdfe1c3dfc77de6/aten/src/ATen/TensorIndexing.cpp" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/1494005cfdf228f8441c1a7aecdfe1c3dfc77de6/aten/src/ATen/TensorIndexing.cpp</a></h4>
<pre><code class="lang-cpp">#include &lt;ATen/TensorIndexing.h&gt;

#include &lt;c10/util/Exception.h&gt;

namespace at {
namespace indexing {

const EllipsisIndexType Ellipsis = EllipsisIndexType();

std::ostream&amp; operator&lt;&lt;(std::ostream&amp; stream, const Slice&amp; slice) {
  stream &lt;&lt; slice.start() &lt;&lt; ":" &lt;&lt; slice.stop() &lt;&lt; ":" &lt;&lt; slice.step();
  return stream;
}

std::ostream&amp; operator&lt;&lt;(std::ostream&amp; stream, const TensorIndex&amp; tensor_index) {
  if (tensor_index.is_none()) {
    stream &lt;&lt; "None";
  } else if (tensor_index.is_ellipsis()) {
    stream &lt;&lt; "...";
  } else if (tensor_index.is_integer()) {
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/1494005cfdf228f8441c1a7aecdfe1c3dfc77de6/aten/src/ATen/TensorIndexing.cpp" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Tensor Tensor::index(ArrayRef<a>at::indexing::TensorIndex</a> indices) const {<br>
TORCH_CHECK(indices.size() &gt; 0, “Passing an empty index list to Tensor::index() is not valid syntax”);<br>
OptionalDeviceGuard device_guard(device_of(*this));<br>
return at::indexing::get_item(*this, indices);<br>
}</p>
<p>I suspect the device guard is heavy. Further, the get_item call is here:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/09660896c0dd2bec888857300a7be9edb52dd05d/aten/src/ATen/TensorIndexing.h" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/09660896c0dd2bec888857300a7be9edb52dd05d/aten/src/ATen/TensorIndexing.h" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/09660896c0dd2bec888857300a7be9edb52dd05d/aten/src/ATen/TensorIndexing.h</a></h4>
<pre><code class="lang-h">#pragma once

#include &lt;c10/util/Optional.h&gt;
#include &lt;ATen/core/TensorBody.h&gt;
#include &lt;ATen/ExpandUtils.h&gt;
#include &lt;ATen/Functions.h&gt;

namespace at {
namespace indexing {

const int64_t INDEX_MAX = std::numeric_limits&lt;int64_t&gt;::max();
const int64_t INDEX_MIN = std::numeric_limits&lt;int64_t&gt;::min();

enum class TensorIndexType { None, Ellipsis, Integer, Boolean, Slice, Tensor };

constexpr c10::nullopt_t None{c10::nullopt_t::init()};

struct CAFFE2_API EllipsisIndexType final { EllipsisIndexType() {} };
CAFFE2_API extern const EllipsisIndexType Ellipsis;

</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/09660896c0dd2bec888857300a7be9edb52dd05d/aten/src/ATen/TensorIndexing.h" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>self_device is even pass around. My guess is that once CPU or GPU device stuff is involved. It is going to be expensive.</p>