Torch.norm 3-6x slower than manually calculating sum of squares?

jpopham91: <p>I just happened to notice that computing the L2 distance between two tensors is consistently 3-6 times slower when using the builtin torch.norm versus manually computing the square root of the sum of squares.</p>
<pre><code class="lang-auto">x = torch.randn(1024, 256)
y = torch.randn(1024, 256)
</code></pre>
<pre><code class="lang-auto">%%timeit
torch.sqrt((x - y).pow(2).sum(1))
</code></pre>
<p><code>333 µs ± 2.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)</code></p>
<pre><code class="lang-auto">%%timeit
torch.norm(x - y, 2, 1)
</code></pre>
<p><code>2.01 ms ± 102 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)</code></p>
<p>Is this to be expected? Does anyone know why there is such a difference in performance here?</p>

richard: <p>That doesn’t sound good… I’ve opened an issue for it here: <a href="https://github.com/pytorch/pytorch/issues/5671">https://github.com/pytorch/pytorch/issues/5671</a></p>

Latope2-150: <p>I ran a few test on GPUs (Titan Xp, 1080Ti and Tesla P100), did not observe this phenomenon. I got very similar timings on GPU for both functions. I also tried it with larger tensors.</p>

richard: <p><a class="mention" href="/u/latope2-150">@Latope2-150</a> do you notice this phenomenon on the CPU?</p>

Latope2-150: <p>Yes, same performance issue on CPU in term of magnitude. Tried it with my i7-7600U on a laptop and with an E5-2680 (but I was not the only one using the server so it might not be a fair test). I noticed the difference is even more important when the tensor are smaller.</p>

vikramnitin9: <p>I still have this problem on the CPU (although not on the GPU). I have compiled the latest version of PyTorch from source (commit <code>#542c273</code>) with MKL-DNN. <code>torch.norm</code> (L2) seems to be about 3-5 times slower than a square-root of a sum of squares.</p>
<p>Here is my code to reproduce this. On my system, this code takes <code>383.1</code> seconds with line <span class="hashtag">#1</span>, and <code>75.2</code> seconds with line <span class="hashtag">#2</span>.</p>
<pre><code class="lang-auto">import torch
import time

loss = 0

A = torch.rand(10, 100, 1000)
A.requires_grad = True

start = time.time()

for i in range(0, 20000):

	B = torch.norm(A, dim=0)  #1

	# B = torch.sqrt(torch.sum(A * A, dim = 0))  #2

	loss += torch.sum(B)

loss.backward()
end = time.time()

print(end - start)
</code></pre>

Aidas_Liaudanskas: <p>May 29th 2019, running PyTorch 1.1.0 on Intel® Core™ i7-6920HQ CPU @ 2.90GHz</p>
<p>The difference persists, get ~13x slowdown on ~14,000 length vector:</p>
<pre><code class="lang-auto">%%timeit
distances = torch.norm(vertices - point_locs, p=2, dim=1)
3.63 ms ± 43.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">%%timeit
distances = torch.sqrt((vertices - point_locs).pow(2).sum(1))
271 µs ± 5.87 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
</code></pre>

sakaia: <p>I also report it. It is easily reproduce on Colab problem still remain.</p>
<pre><code class="lang-auto">x = torch.randn(1024, 256)
y = torch.randn(1024, 256)
%timeit torch.sqrt((x - y).pow(2).sum(1))
%timeit torch.norm(x - y, 2, 1)
x = x.cuda()
y = y.cuda()
%timeit torch.sqrt((x - y).pow(2).sum(1))
%timeit torch.norm(x - y, 2, 1)
</code></pre>
<p>results</p>
<pre><code class="lang-auto">1000 loops, best of 3: 465 µs per loop
100 loops, best of 3: 4.23 ms per loop
The slowest run took 5.25 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 44 µs per loop
The slowest run took 4.03 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 24 µs per loop
</code></pre>