C++ is slower than corresponding py version

username: 7starsea
cooked: <p>I have developed a batch linear regression algorithm in c++ which reads</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;
/// param x: (batch, sample, feature)
/// param y: (batch, sample)
/// return (batch, feature)
at::Tensor lr_batch_fit(const at::Tensor &amp;x, const at::Tensor &amp;y){
    const int64_t batch = x.size(0), sample = x.size(1), features = x.size(2);

    const at::Tensor &amp; xt = at::transpose(x, 1, 2);
    const at::Tensor &amp; s = at::bmm(xt, x);
    const at::Tensor &amp; yp = y.view({batch, sample, 1});
    const at::Tensor &amp; pre_beta = at::bmm(xt, yp);

    at::Tensor s_lu, pivots, infos;
    std::tie(s_lu, pivots, infos) = at::_lu_with_info(s, true, true);
    /*
    const auto data = at::_lu_with_info(s, true, true);
    const at::Tensor &amp; s_lu = std::get&lt;0&gt;(data);
    const at::Tensor &amp; pivots = std::get&lt;1&gt;(data);
    const at::Tensor &amp; infos = std::get&lt;2&gt;(data);
    */

    at::Tensor beta = at::zeros({batch, features, 1}, x.options());
    if(0 == infos.nonzero().size(0)){
        at::lu_solve_out(beta, pre_beta, s_lu, pivots);
    }
    return beta.squeeze(2);
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.def("lr_batch_fit_c",   &amp;lr_batch_fit);
}
</code></pre>
<p>The corrresponding python code is</p>
<pre><code class="lang-auto">import torch

def lr_batch_fit_py(x, y):
    """
    :param x: (batch, sample, features)
    :param y: (batch, sample,)
    :return: beta (batch, features)
    """
    batch, sample, features = x.shape
    xt = torch.transpose(x, 1, 2)
    s = torch.bmm(xt, x)
    y = y.view(batch, sample, 1)
    pre_beta = torch.bmm(xt, y)
    s_lu, pivots, infos = torch.lu(s, pivot=True, get_infos=True)

    beta = torch.zeros(batch, features, 1, dtype=x.dtype, device=x.device)
    if 0 == infos.nonzero().size(0):
        torch.lu_solve(pre_beta, s_lu, pivots, out=beta)
    return beta.squeeze(2)
</code></pre>
<p>and here is the testing code</p>
<pre><code class="lang-auto">import torch
import time
from lr_batch_c import lr_batch_fit_c
from lr_batch import lr_batch_fit_py

def test_lr_batch():
    def check_equal(x, y, eps=1e-8):
        assert torch.sum(torch.abs(x - y)) &lt;= eps

    tt1, tt2 = 0, 0
    x = torch.randn(4, 48000, 160, dtype=torch.float64).cuda()
    beta = torch.randn(4, 160, 1, dtype=torch.float64).cuda()
    y = torch.bmm(x, beta).squeeze(2)
    beta = beta.squeeze(2)
    for i in range(100):
        t1 = time.time()
        beta2 = lr_batch_fit_c(x, y)
        t2 = time.time()
        beta3 = lr_batch_fit_py(x, y)
        t3 = time.time()
        tt1 += (t2 - t1)
        tt2 += (t3 - t2)
        check_equal(beta, beta2)
        check_equal(beta, beta3)
    print("test_lr_batch  c_time %.2fs py_time %.2fs" % (tt1, tt2))
</code></pre>
<p>After running <code>test_lr_batch</code> several times with cuda, I found that the c++ version is slower than the py version. In my naive thinking, since c++ version will save some time at the boundary between py and c++, it should be faster. Could anyone help me figure it out?<br>
(Note with cpu, the c++ version is a little bit faster than py verion, as anticipated.)<br>
Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you’re runnning on cuda, you need to add the appropriate <code>torch.cuda.synchronize()</code> to make your timings accurate (keep in mind that all the CUDA api is asynchronous).<br>
Also how big a difference do you see?</p>

username: 7starsea
cooked: <p>Hi,<br>
my testing result is</p>
<pre><code class="lang-auto">test_lr_batch  c_time 1.54s py_time 1.20s
</code></pre>
<p>If i change x to <code>torch.randn(4, 4800, 160)</code>, the result is</p>
<pre><code class="lang-auto">test_lr_batch  c_time 0.57s py_time 0.24s
</code></pre>
<p>and it seems the difference is (somehow) significant.<br>
(by the way, i am using pytorch 1.4 + ( NVIDIA-SMI 430.40 Driver Version: 430.40 CUDA Version: 10.1) + gcc 7.5.0 + Ubuntu 18.04.4 LTS)</p>
<p>I added <code>torch.cuda.synchronize()</code> at different locations in the function <code>test_lr_batch</code> and it makes no change to the results.</p>

username: pascal.soveaux
cooked: <p>Hello,</p>
<p>Maybe a stupid question : are you sure your C++ app runs on CUDA? Release mode?</p>
<p>I am surprised because my C++ app runs much faster than python (but that’s for many different reasons).</p>
<p>Pascal</p>

username: 7starsea
cooked: <p>Thanks for your kind reminder and here are my double checks.<br>
First, the setup file is</p>
<pre><code class="lang-auto">from setuptools import setup
from torch.utils.cpp_extension import CppExtension, BuildExtension

setup(name='lr_batch_c',
      ext_modules=[CppExtension('lr_batch_c', ['lr_batch.cpp'])],
      cmdclass={'build_ext': BuildExtension})
</code></pre>
<p>and I saw flags <code>-DNDEBUG -O3</code> when using <code>python setup.py build_ext --inplace</code> to compile the cpp file.</p>
<p>Second, except the explicit transferring of x, y to cuda, all other operations are using standard pytorch api’s and those operations should runs on CUDA, right?</p>

username: 7starsea
cooked: <p>Step by step testing, it looks like the cpp version of lu</p>
<pre><code class="lang-auto">at::Tensor s_lu, pivots, infos;
    std::tie(s_lu, pivots, infos) = at::_lu_with_info(s, true, true);
</code></pre>
<p>is slower than the py version</p>
<pre><code class="lang-auto">    s_lu, pivots, infos = torch.lu(s, pivot=True, get_infos=True)
</code></pre>

username: albanD
cooked: <p>You need to put a <code>torch.cuda.synchronize()</code> before every call to <code>time.time()</code> to get actuall timings. Otherwise you’ll just see random artifacts from the time it takes to queue stuff on the GPU and when that queue is full.</p>

username: 7starsea
cooked: <p>Here is my new <code>test_lr_batch</code> function</p>
<pre><code class="lang-auto">def test_lr_batch():
    def check_equal(x, y, eps=1e-8):
        assert torch.sum(torch.abs(x - y)) &lt;= eps

    tt1, tt2 = 0, 0
    x = torch.randn(4, 4800, 160, dtype=torch.float64).cuda()
    beta = torch.randn(4, 160, 1, dtype=torch.float64).cuda()
    y = torch.bmm(x, beta).squeeze(2)
    beta = beta.squeeze(2)
    torch.cuda.synchronize()
    for i in range(200):
        torch.cuda.synchronize()
        t1 = time.time()

        beta2 = lr_batch_fit_c(x, y)

        torch.cuda.synchronize()
        t2 = time.time()

        beta3 = lr_batch_fit_py(x, y)

        torch.cuda.synchronize()
        t3 = time.time()
        tt1 += (t2 - t1)
        tt2 += (t3 - t2)
        check_equal(beta, beta2)
        check_equal(beta, beta3)
    print("test_lr_batch  c_time %.3fs py_time %.3fs" % (tt1, tt2))
</code></pre>
<p>Based on the new function, the cpp version is still slower than the py version.</p>
<p>Is the new function fine for testing, or should I also put similar stuff ( <code>torch.cuda.synchronize()</code> ) in the cpp code?</p>

username: glaringlee
cooked: <p>Can you test your c++ code under c++ env without binding? (use libtorch)<br>
The python apis and c++ apis actually hit the same underlying c++ code, the slowness is probably due to python binding for cpp extension.</p>

username: 7starsea
cooked: <p>I don’t think the slowness is due to the python binding since I tested the following c++ code</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;
/// param x: (batch, sample, feature)
/// param y: (batch, sample)
/// return (batch, feature)
at::Tensor lr_batch_fit(const at::Tensor &amp;x, const at::Tensor &amp;y){
    const int64_t batch = x.size(0), sample = x.size(1), features = x.size(2);

    const at::Tensor &amp; xt = at::transpose(x, 1, 2);
    const at::Tensor &amp; s = at::bmm(xt, x);
    const at::Tensor &amp; yp = y.view({batch, sample, 1});
    const at::Tensor &amp; pre_beta = at::bmm(xt, yp);

    at::Tensor beta = at::zeros({batch, features, 1}, x.options());
    return beta.squeeze(2);
</code></pre>
<p>and the corresponding python version</p>
<pre><code class="lang-auto">import torch

def lr_batch_fit_py(x, y):
    """
    :param x: (batch, sample, features)
    :param y: (batch, sample,)
    :return: beta (batch, features)
    """
    batch, sample, features = x.shape
    xt = torch.transpose(x, 1, 2)
    s = torch.bmm(xt, x)
    y = y.view(batch, sample, 1)
    pre_beta = torch.bmm(xt, y)
    beta = torch.zeros(batch, features, 1, dtype=x.dtype, device=x.device)
    return beta.squeeze(2)
</code></pre>
<p>both versions have mildly difference and c++ wins the race.</p>

username: glaringlee
cooked: <p>Hi, I think there are some difference between torch.lu(s, pivot=True, get_infos=True) and at::_lu_with_info(s, true, true)</p>
<p>check here:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/tensor.py#L333" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/tensor.py#L333" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/tensor.py#L333</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="323" style="counter-reset: li-counter 322 ;">
<li>    else:</li>
<li>        return self.flip(0)</li>
<li>
</li>
<li>def norm(self, p="fro", dim=None, keepdim=False, dtype=None):</li>
<li>    r"""See :func:`torch.norm`"""</li>
<li>    return torch.norm(self, p, dim, keepdim, dtype=dtype)</li>
<li>
</li>
<li>def lu(self, pivot=True, get_infos=False):</li>
<li>    r"""See :func:`torch.lu`"""</li>
<li>    # If get_infos is True, then we don't need to check for errors and vice versa</li>
<li class="selected">    LU, pivots, infos = torch._lu_with_info(self, pivot=pivot, check_errors=(not get_infos))</li>
<li>    if get_infos:</li>
<li>        return LU, pivots, infos</li>
<li>    else:</li>
<li>        return LU, pivots</li>
<li>
</li>
<li>def stft(self, n_fft, hop_length=None, win_length=None, window=None,</li>
<li>         center=True, pad_mode='reflect', normalized=False, onesided=True):</li>
<li>    r"""See :func:`torch.stft`</li>
<li>
</li>
<li>    .. warning::</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>If get_infos is True, then check_error flag in at version should be false. So you should use at::_lu_with_info(s, true, false) in the test.</p>
<p>Let me know if the performance goes back to normal after this change.</p>

username: 7starsea
cooked: <p>Thanks for pointing out the difference, but the performance remains the same after this change, so sad.</p>

username: albanD
cooked: <p>Note that as the heavy lifting is done by the exact same function, the only thing you measure is the overhead of calling the function.<br>
And while python overhead is significant if you loop over very simple ops that take micro seconds. If your function actually does stuff, you won’t even see it.</p>