Speed-up libtorch operators on CPU

username: Joe_Harrison
cooked: <p>I’ve done a speed comparison between simple operators in libtorch and simple vectors. I don’t know entirely whether the comparison is fair, but the vector method seems to be about 60 times faster. Is there a way to improve speed on simple operators? I want to know whether it would make sense to convert back and forth from tensor to vector and back during inference on the CPU.</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
    Tensor X = torch::randn({ 253 });
    Tensor output = X;

    std::chrono::time_point&lt;std::chrono::system_clock&gt; time = chrono::system_clock::now();
    {
        torch::NoGradGuard no_grad;
        for (int i = 0; i &lt; 10000000; i++) {
            output = X * X;
        }
    }

    cout&lt;&lt;chrono::duration_cast&lt;chrono::milliseconds&gt;(chrono::system_clock::now() - time).count()/1000.&lt;&lt;endl;
    //Convert tensor to vector
    std::vector&lt;float&gt; v(X.data_ptr&lt;float&gt;(), X.data_ptr&lt;float&gt;() + X.numel());
    vector&lt;float&gt; results = v;

    time = chrono::system_clock::now();
    for(int i=0;i&lt;10000000;i++){

        std::transform(v.cbegin(), v.cend(), v.cbegin(), results.begin(), std::multiplies&lt;float&gt;());
    }
    cout&lt;&lt;chrono::duration_cast&lt;chrono::milliseconds&gt;(chrono::system_clock::now() - time).count()/1000.&lt;&lt;endl;
</code></pre>
<p>Libtorch: 7.125 seconds<br>
Standard vector: 0.282 seconds</p>

username: ryanleary
cooked: <p>Which compiler are you using? Are you sure that loop isn’t being optimized out? I see the following on my system:</p>
<pre><code class="lang-auto">15.365
51.345
</code></pre>

username: googlebot
cooked: <p>if you have loops with 100+ iterations in practice, then yes, you should be using data_ptr&lt;T&gt; for “unboxing”. vector is not the best choice here, instead you can either work with pointers or use something like Eigen arrays (mapped).</p>

username: googlebot
cooked: <p>and I forgot about accessors. this is from docs:</p>
<pre><code class="lang-auto">torch::Tensor foo = torch::rand({12, 12});

// assert foo is 2-dimensional and holds floats.
auto foo_a = foo.accessor&lt;float,2&gt;();
float trace = 0;

for(int i = 0; i &lt; foo_a.size(0); i++) {
  // use the accessor foo_a to get tensor data.
  trace += foo_a[i][i];
}
</code></pre>

username: Joe_Harrison
cooked: <p>/usr/bin/c++ --version gives me<br>
c++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0</p>

username: Eugene7
cooked: <p>To help you train the <em>faster</em> , here are 8 tips you should be aware of that … data from GPU to <em>CPU</em> and dramatically slows your performance. As you can see, the models using NNAPI run about 25-30% <em>faster</em> for both Float32 and Int8 compared with the <em>CPU</em> models.</p>

username: Joe_Harrison
cooked: <p>I’m not transferring any data from GPU to CPU, I’m just using the CPU. Working with many tiny networks (32 nodes max) in parallel</p>

username: Joe_Harrison
cooked: <p>The actual problem isn’t a loop of matrix operations, I just used this to time the operation. I don’t want to use the vector implementation because then I’d have to write backpropagation myself. My networks are actually very small consisting of just 4 basic vector operations (+,-,x,/). Everything is run on the CPU.</p>

username: googlebot
cooked: <p>but the big absolute overhead you’ve observed is from repeated Tensor unwrappings in the loop. If your neural network iteration consists of 100-1000 operations, and tensor sizes are bigger (from batching), this overhead is not so significant.</p>
<p>for long sequences of piecewise operations, JIT compiling does a decent job fusing them (it used to need USE_CPU_FUSER=1 env. variable, haven’t tried it recently)</p>

username: dtsaras
cooked: <p>Hey, did you end up finding out how to speed up this process?</p>

username: Joe_Harrison
cooked: <p>Yes, but not satisfyingly. I ended up writing my own backprop. Torch is great for larger models where the overhead makes sense</p>