Torch.where got 10x slower than normal

youzunzhi: <p>I’m pretty sure it’s about my device, probably the cpu. Everything else seems fine, but one day <code>torch.where</code> became really slow.</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)
import time
t = torch.randint(0,100,(1000,1000))
time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
</code></pre>
<p>the result is:</p>
<pre><code class="lang-auto">1.6.0
torch.where time: 0.041625261306762695
</code></pre>
<p>but in my own MacBook Pro, it is much faster:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
... print(torch.__version__)
... import time
... t = torch.randint(0,100,(1000,1000))
... time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
... 

1.6.0
torch.where time: 0.0031599998474121094
</code></pre>
<p>Also to mention that the time may vary, but it is always longer than 0.01s:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.014759302139282227
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.06390643119812012
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.019058942794799805
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.041625261306762695
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.03616046905517578
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.0942380428314209
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.09639406204223633
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.09022760391235352
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.019794464111328125
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.08239603042602539
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.015592098236083984
&gt;&gt;&gt; time_ = time.time();torch.where(t == 0);print('torch.where time:', time.time() - time_)
(tensor([  0,   0,   0,  ..., 999, 999, 999]), tensor([ 65, 254, 354,  ..., 937, 971, 984]))
torch.where time: 0.024471759796142578
</code></pre>
<p>Any idea about where I should check for the device of the server I’m using? Many thanks.</p>

BramVanroy: <p>First things first: using <code>time.time</code> on a single run is not a good benchmark. You’re better off using built-in functions that are meant for this kind of thing. Have a look at the <a href="https://docs.python.org/3.8/library/timeit.html" rel="noopener nofollow ugc"><code>timeit</code></a> module.</p>
<p>Second, it is not really clear what you are comparing here. You say that things “got slower than usual” but then compare with a Macbook. So what are you comparing? Different versions of Pytorch on the macbook? Different computers altogether? Please give more information.</p>

youzunzhi: <p>Apologies for not being clear. I’m comparing the running speed of the server I’m using with the normal speed of this line of code, where I was assuming that the speed on my MacBook is the “normal speed”. Sorry for that. But I do believe that 0.01s is unusual, probably 10x slower than any other normally-working device. As I said, one day it suddenly need 10x more time to execute my code. But I couldn’t figure out what went wrong.</p>
<p>Thank you for your advice of using <code>timeit</code>. I tried and here are the results:<br>
On my slow server;</p>
<pre><code class="lang-auto">&gt;&gt;&gt; timeit.timeit('import torch;torch.where(torch.randint(0,100,(1000,1000)) == 0)', number=100)
18.391881837975234
</code></pre>
<p>On my Macbook:</p>
<pre><code class="lang-auto">timeit.timeit('import torch;torch.where(torch.randint(0,100,(1000,1000)) == 0)', number=100)
1.2651853299998947
</code></pre>

BramVanroy: <p>Although it is true that in single threaded mode, regular CPUs are often faster (higher clock speeds), the difference in your example is very large. I tested locally (i7-8700k @ 4.5GHz; 1.01s) and on our server (Xeon Gold 6142 CPU @ 2.60GHz; 2.16s) and the difference is not at all that noticeable. Some things you can check:</p>
<ul>
<li>same version of PyTorch and Python (should not matter, but might indicate a version bug)</li>
<li>make sure the server is not busy (check <code>htop</code> for instance)</li>
<li>if you system is doing a lot IO (reading/writing) at the time that you run the benchmark, that can also negatively impact your CPU throughput</li>
</ul>