Cdist vs matmul

username: JieLei
cooked: <p>Hi,</p>
<p>I am trying to build a video retrieval system using cosine similarity. L2 distance could also be used as it could be written as <code>|| a - b || = 2 - 2 * &lt;a, b&gt;</code>, where <code>a</code>, <code>b</code> are both normalized vectors.</p>
<p>Now I have two matrice <code>A: [N x d]</code>, <code>B: [M x d]</code></p>
<p>L2 distance can be calculated in PyTorch as <code>torch.pdist(A, B)</code>, cosine similarity as inner product <code>torch.mm(A, B.transpose(0, 1))</code>. However, I found later to be much slower than the former. Any idea why?</p>
<p>Below is the code I used to do the comparison.</p>
<pre><code class="lang-auto">import time
import torch
import torch.nn.functional as F
import numpy as np

def compare_l2dist_inner_product_time(n_videos=2000, d=256, n_query=1000, n_runs=5):
    st_time = time.time()
    fake_database = F.normalize(torch.randn((n_videos, d), dtype=torch.float32).cuda(), dim=1, p=2)
    fake_query = F.normalize(torch.randn((n_query, d), dtype=torch.float32).cuda(), dim=1, p=2)
    print("Construct fake database + query time {}".format(time.time() - st_time))
    print("fake_database shape {} fake_query shape {}".format(fake_database.shape, fake_query.shape))

    times_l2dist = []
    for _ in range(n_runs):
        st_time = time.time()
        l2_dist = torch.cdist(fake_query, fake_database, p=2)  # (n_query, n_videos)
        times_l2dist.append(time.time() - st_time)
    avg_time_l2dist = np.mean(times_l2dist)
    print("L2 Distance time {}".format(avg_time_l2dist))

    times_ip = []
    fake_database = fake_database.transpose(0, 1)
    for _ in range(n_runs):
        st_time = time.time()
        inner_product = torch.mm(fake_query, fake_database)  # (n_query, n_videos)
        times_ip.append(time.time() - st_time)
    avg_time_ip = np.mean(times_ip)
    print("Inner Product time {}".format(avg_time_ip))

compare_l2dist_inner_product_time()
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Construct fake database + query time 7.20833158493042
fake_database shape torch.Size([2000, 256]) fake_query shape torch.Size([1000, 256])
L2 Distance time 5.9604644775390625e-05
Inner Product time 0.07725939750671387
</code></pre>

username: JieLei
cooked: <p>OK I figured it out, seems PyTorch need some time to allocate cuda memory.</p>

username: ptrblck
cooked: <p>Not only this, but CUDA operations are executed asynchronously, so you should synchronize before starting and stopping the timer using <code>torch.cuda.synchronize()</code>.</p>

username: JieLei
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="61682">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.cuda.synchronize()</p>
</blockquote>
</aside>
<p>Thanks! Added <code>torch.cuda.synchronize()</code> as <a class="mention" href="/u/ptrblck">@ptrblck</a> suggested, also added some warmup runs to make sure it does not count in memory allocation time.</p>
<pre><code class="lang-auto">def compare_l2dist_inner_product_time(n_videos=2000, d=256, n_query=1000, n_runs=10, n_warmup_runs=10):
    torch.cuda.synchronize()
    st_time = time.time()
    fake_database = F.normalize(torch.randn((n_videos, d), dtype=torch.float32).cuda(), dim=1, p=2)
    fake_query = F.normalize(torch.randn((n_query, d), dtype=torch.float32).cuda(), dim=1, p=2)
    torch.cuda.synchronize()
    print("Construct fake database + query time {}".format(time.time() - st_time))
    print("fake_database shape {} fake_query shape {}".format(fake_database.shape, fake_query.shape))

    times_l2dist = []
    for _ in range(n_warmup_runs + n_runs):
        torch.cuda.synchronize()
        st_time = time.time()
        l2_dist = torch.cdist(fake_query, fake_database, p=2)  # (n_query, n_videos)
        torch.cuda.synchronize()
        times_l2dist.append(time.time() - st_time)
    avg_time_l2dist = np.mean(times_l2dist[n_warmup_runs:])
    print("L2 Distance time {}".format(avg_time_l2dist))

    times_ip = []
    fake_database = fake_database.transpose(0, 1)
    for _ in range(n_warmup_runs + n_runs):
        torch.cuda.synchronize()
        st_time = time.time()
        inner_product = torch.mm(fake_query, fake_database)  # (n_query, n_videos)
        torch.cuda.synchronize()
        times_ip.append(time.time() - st_time)
    avg_time_ip = np.mean(times_ip[n_warmup_runs:])
    print("Inner Product time {}".format(avg_time_ip))

</code></pre>
<p>Here are the outputs (On an RTX 2080Ti):</p>
<pre><code class="lang-auto">Construct fake database + query time 0.008526802062988281
fake_database shape torch.Size([2000, 256]) fake_query shape torch.Size([1000, 256])
L2 Distance time 0.014820019404093424
Inner Product time 0.00018693606058756512
</code></pre>
<p>The results look more reasonable now, though I did not expect Inner Product to be so fast compared to <code>torch.cdist</code>.</p>

username: ptrblck
cooked: <p>Thanks for the code! I wouldn’t expect such a difference, so I just rerun the code and got:</p>
<pre><code class="lang-python">Construct fake database + query time 0.018206119537353516
fake_database shape torch.Size([2000, 256]) fake_query shape torch.Size([1000, 256])
L2 Distance time 0.00036679506301879884
Inner Product time 0.0001362919807434082
</code></pre>
<p>on a TitanV.</p>

username: JieLei
cooked: <p>That’s strange. I tried on a Titan X (Pascal) and 1080Ti and got similar results as my 2080Ti. I am using Python 3.7, PyTorch 1.3.1 with cuda 10, which versions are you using?</p>

username: ptrblck
cooked: <p>I’m using <code>1.4.0.dev20191109</code> with Python3.7 and CUDA10.1.</p>

username: JieLei
cooked: <p>After using <code>1.4.0.dev20191109 with Python3.7 and CUDA10.1.</code> I get a similar performance. It seems like a PyTorch1.3.1 or CUDA 10.0 problem.</p>