PyTorch Tensor slow on small matrix operations (large overhead?)

username: JohannesB
cooked: <p>For small matrix operations on CPU (e.g., 3x3 matrices), NumPy seems to be approximately 15x faster than PyTorch. This is relevant for operations in the data loader or (geometric) transformations where I could achieve great speedups by using NumPy arrays instead tensors.</p>
<p>Here is a small performance test which just adds a constant to a 3x3 matrix:</p>
<pre><code class="lang-auto">&gt;python -mtimeit -s"import numpy as np; test = np.random.rand(3,3)" "test + 1"
500000 loops, best of 5: 765 nsec per loop

&gt; python -mtimeit -s "import torch; test = torch.rand((3, 3))" "test + 1"
20000 loops, best of 5: 11.2 usec per loop
</code></pre>
<p>I assume that the overhead for PyTorch tensors is much higher than for NumPy arrays. When using larger arrays and tensors, the overhead will be less relevant and I have seen contributions in this forum and on GitHub where the performances seem to be comparable.</p>
<p>I am working with PyTorch 1.3, NumPy 1.16.4 and Python 3.7.4 on a 64 bit Windows machine.</p>
<p>(Btw: I found some articles in this forum comparing the speed of <em>specific</em> operators and discussing the processing speed of <em>large</em> matrices but not about the <em>overhead</em> of the matrix classes itself. If I have missed an article – please let me know!)</p>

username: ptrblck
cooked: <p>I would assume PyTorch might have a larger overhead, but I’m getting similar numbers for your test:</p>
<pre><code class="lang-python">%timeit "import numpy as np; test = np.random.rand(3,3)" "test + 1"
&gt; 6.31 ns ± 0.188 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each)

%timeit "import torch; test = torch.rand((3, 3))" "test + 1"
&gt; 6.12 ns ± 0.0184 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each)
</code></pre>
<p>for PyTorch <code>1.5.0.dev20200122</code>.</p>

username: JohannesB
cooked: <p>Did you get your results on a Linux/MacOS platform?</p>
<p>We tested the nightly Windows build <code>1.5.0.dev20200128</code>, but the PyTorch tensor version still seems to be much slower than the NumPy array version. Could this be a platform-dependent problem?</p>
<p>Is there anyone who could run the small performance tests on a Windows platform as well?</p>

username: ptrblck
cooked: <p>I used Ubuntu18.04. Sorry for not mentioning it.</p>

username: G.M
cooked: <p>I’m using Pytorch 1.4.0, numpy 1.16.3 with Python 3.7.3 on windows 10 64 bit.</p>
<pre><code class="lang-auto">&gt;python -mtimeit -s"import numpy as np; test = np.random.rand(3,3)" "test + 1"
500000 loops, best of 5: 713 nsec per loop
</code></pre>
<pre><code class="lang-auto">&gt;python -mtimeit -s "import torch; test = torch.rand((3, 3))" "test + 1"
50000 loops, best of 5: 5.14 usec per loop
</code></pre>

username: peterjc123
cooked: <p>Result on Windows 10, Python 3.7.4 (Anaconda), NumPy 1.16.5, PyTorch 1.4.0.</p>
<blockquote>
<p>python -mtimeit -s “import torch; test = torch.rand((3, 3))” “test + 1”<br>
20000 loops, best of 5: 9.86 usec per loop</p>
</blockquote>
<blockquote>
<p>python -mtimeit -s"import numpy as np; test = np.random.rand(3,3)" “test + 1”<br>
500000 loops, best of 5: 834 nsec per loop</p>
</blockquote>
<p>I guess the reason is that some simd clauses are not enabled for MSVC.</p>

username: ptrblck
cooked: <p>I’m not sure why, but IPython’s <code>%timeit</code> command returns the initial low timings, while running the <code>python -mtimeit</code> cmd directly in the terminal returns:</p>
<pre><code class="lang-python">python -mtimeit -s"import numpy as np; test = np.random.rand(3,3)" "test + 1"
500000 loops, best of 5: 894 nsec per loop

python -mtimeit -s "import torch; test = torch.rand((3, 3))" "test + 1"
50000 loops, best of 5: 4.9 usec per loop
</code></pre>
<p>so comparable to what others are seeing.<br>
I’m not sure, which command to trust more and maybe someone knows, what’t the best way to time on the CPU.</p>

username: JohannesB
cooked: <p>I would trust the terminal <code>python -mtimeit</code> results since they reflect my experiences in my training framework: By replacing small PyTorch matrix operations by NumPy operations, I could achieve a great speed-up. (This mainly affects data loading and augmentation, which is not vectorizable in my case. The large training vector operations are of course carried out on the GPU.)</p>
<p><a class="mention" href="/u/peterjc123">@peterjc123</a>: I do not think that this has something to do with SIMD instructions since both of the above tests run on a single CPU as my task monitor indicates. Additionally, you could re-run the tests with a 1x1 matrix yielding approximately the same results.</p>
<p>Here is an alternative timing measurement delivering similar results:</p>
<pre><code class="lang-auto">import torch
import time
iterations = 10000
test_tensor = torch.zeros((1, 1))
test_array = test_tensor.numpy().copy()
time_start = time.time()
for i in range(iterations):
    test_tensor = test_tensor + 1
t_torch = (time.time() - time_start) / iterations
time_start = time.time()
for i in range(iterations):
    test_array = test_array + 1
t_numpy = (time.time() - time_start) / iterations
print('Time elapsed (PyTorch tensor): {0:.3f} µs'.format(t_torch * 1E6))
print('Time elapsed (NumPy array): {0:.3f} µs'.format(t_numpy * 1E6))

Time elapsed (PyTorch tensor): 9.798 µs
Time elapsed (NumPy array): 0.800 µs
</code></pre>