Torch forward very slow

craftyvisage: <p>For some reason running my forward implementation in torch is very slow. See simple example below. When I run this without the net-&gt;forward loop, it runs in 0.132s. With that loop included it runs in 7.694s. What is going on here?</p>
<p>My CMakeLists.txt is also below and I installed torch from the LibTorch ZIP archive. Do I need to build it myself?</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;vector&gt;
#include &lt;torch/torch.h&gt;

struct Net : torch::nn::Module {
    Net(int input_size, int layer_width) {
	linear1 = register_module("linear1",
				  torch::nn::Linear(input_size, layer_width));
	linear2 = register_module("linear2",
				  torch::nn::Linear(layer_width, layer_width));
	linear3 = register_module("linear3",
				  torch::nn::Linear(layer_width, layer_width));
	// set weights
	std::vector&lt;float&gt; weights = {0.1, 0.2, 0.3};
	std::vector&lt;float&gt; biases = {0.4, 0.4, 0.4};
	// layer 1
	linear1-&gt;bias[0] = biases[0];
	linear1-&gt;bias[1] = biases[1];
	linear1-&gt;bias[2] = biases[2];
 	linear1-&gt;weight[0] = torch::from_blob(weights.data(),
					      layer_width).clone();
	linear1-&gt;weight[1] = torch::from_blob(weights.data(),
					      layer_width).clone();
	linear1-&gt;weight[2] = torch::from_blob(weights.data(),
					      layer_width).clone();
	// layer 2
	linear2-&gt;bias[0] = biases[0];
	linear2-&gt;bias[1] = biases[1];
	linear2-&gt;bias[2] = biases[2];
	linear2-&gt;bias = torch::from_blob(biases.data(),
					 layer_width).clone();
	linear2-&gt;weight[0] = torch::from_blob(weights.data(),
					      layer_width).clone();
	linear2-&gt;weight[1] = torch::from_blob(weights.data(),
					      layer_width).clone();
	linear2-&gt;weight[2] = torch::from_blob(weights.data(),
					      layer_width).clone();
	// layer 3
	linear3-&gt;bias[0] = biases[0];
	linear3-&gt;bias[1] = biases[1];
	linear3-&gt;bias[2] = biases[2];
	linear3-&gt;bias = torch::from_blob(biases.data(),
					 layer_width).clone();
	linear3-&gt;weight[0] = torch::from_blob(weights.data(),
					      layer_width).clone();
	linear3-&gt;weight[1] = torch::from_blob(weights.data(),
					      layer_width).clone();
	linear3-&gt;weight[2] = torch::from_blob(weights.data(),
					      layer_width).clone();
    }

    torch::Tensor forward(torch::Tensor input) {
	torch::Tensor output;
	output = torch::relu(linear1(input));
	output = torch::relu(linear2(output));
	output = linear3(output);
	return output;
    }

    torch::nn::Linear linear1{nullptr};
    torch::nn::Linear linear2{nullptr};
    torch::nn::Linear linear3{nullptr};
};

int main() {
    int input_size = 3;
    int layer_width = 3;
    torch::autograd::GradMode::set_enabled(false);
    std::shared_ptr&lt;Net&gt; net = std::make_shared&lt;Net&gt;(input_size, layer_width);

    std::vector&lt;float&gt; before = {0.1, 0.2, 0.3};
    torch::Tensor a = torch::from_blob(before.data(), 3).clone();
    for (int i = 0; i &lt; 100000; i++) {
    	net-&gt;forward(a);
    }
    std::cout &lt;&lt; "Torch output:\n";
    std::cout &lt;&lt; net-&gt;forward(a) &lt;&lt; std::endl;

    std::vector&lt;float&gt; after(3);
    std::vector&lt;float&gt; weights = {0.1, 0.2, 0.3};
    float bias = 0.4;
    for (int i = 0; i &lt; 100000; i++) {
	before = {0.1, 0.2, 0.3};
	for (int layer = 0; layer &lt; 3; layer++) {
	    for (int node = 0; node &lt; 3; node++) {
		after[node] = 0.0;
		for (int weight_idx = 0; weight_idx &lt; 3; weight_idx++) {
		    after[node] += before[weight_idx] * weights[weight_idx];
		}
		after[node] += bias;
		after[node] = std::max(0.0f, after[node]);
	    }
	    before = after;
	}
    }
    std::cout &lt;&lt; "My output:\n";
    for (auto i : after) std::cout &lt;&lt; i &lt;&lt; "\n";
    return 0;
}
</code></pre>
<p>CMakeLists.txt</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(net)

find_package(Torch REQUIRED)

set(CMAKE_CXX_FLAGS "-Wall -Wextra -O2")
add_executable(net src/main.cpp)
target_link_libraries(net "${TORCH_LIBRARIES}")
set_property(TARGET net PROPERTY CXX_STANDARD 11)
</code></pre>

mhubii: <p>well you forward your network 100000 times, why would you do that? That doesnt make the forwarding slow</p>

craftyvisage: <p>It’s very, very slow compared to the roughly equivalent nested loop pure C++ code below the torch loop in the example I posted. And the nested loop stuff doesn’t even use something like Blas which I think would potentially be even faster.</p>
<p>In general I don’t understand pytorch. I want to use a neural network in my C++ application. I get the weights from somewhere else, so no training is involved. I just end up running a bunch of forward calls for inference similar to the example I posted above. Doing this with libtorch is very slow. Even just creating a tensor for the forward input without even running forward slows the code down at least an order of magnitude.</p>
<p>I tried doing the same thing with an equivalent network using a random C++ NN implementation  I found here:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/yixuan/MiniDNN" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/216398?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/yixuan/MiniDNN" target="_blank" rel="nofollow noopener">yixuan/MiniDNN</a></h3>

<p>A header-only C++ library for deep neural networks - yixuan/MiniDNN</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and the code runs something like 750x faster than libtorch. Is this not a good use case for libtorch? Am I missing something?</p>
<p>At this point I’ve tried building from source and also running on a different cluster. Neither changed anything.</p>

mhubii: <p>ah now I got your point. I dont know</p>

craftyvisage: <p>The answer appears to be that Torch has significant overhead for small problems:</p>
<aside class="quote quote-modified" data-post="3" data-topic="45459">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/f14d63/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-is-my-numpy-implementation-8x-faster-than-pytorch-forward-pass/45459/3">Why is my numpy implementation 8x faster than pytorch forward pass?</a> 
  </div>
  <blockquote>
    I ran the profiler as you suggested. First I made the code easier to profile by splitting the pytorch and my numpy runs. See below. 
Not sure what exactly you expected. Nothing jumps out to me other than everything looks slow. The times 13.7 and 8.44 don’t add up to 14.1 s. Not sure what to make of that. 
[image] 
import IPython as ipy
import numpy as np
import torch
import time

class Net(torch.nn.Module):
    def __init__(self, input_size, layer_width):
        super(Net, self).__init__()
    …
  </blockquote>
</aside>

<p>This is good to keep in mind.</p>

ptrblck: <p>I would expect to see some overhead in Python, but not necessarily in C++.<br>
CC <a class="mention" href="/u/goldsborough">@goldsborough</a>: Is this expected as well?</p>

tom: <p>The overhead here isn’t from any computation (there isn’t much computation going on in the example) but from the bookkeeping, i.e. the creation of all the Tensors - memory allocation setting up tensors etc., so this is expected.<br>
In the example even the - considerable smaller - overhead of vector allocations is not accounted for, things are done in-place to avoid allocations etc., so it is much of an apples-to-oranges comparison.<br>
Once you have a non-negligble amount of computation, you find that this overhead is dwarfed by what’s going on in the actual backend routines.</p>
<p>I don’t think that PyTorch needs to hide in terms of speed but there are a few where we’d like things to be faster, see <a href="https://github.com/pytorch/pytorch/issues/13049" rel="nofollow noopener">this great detailed analysis of overheads in LSTMs</a> by <a class="mention" href="/u/richard">@richard</a> for a taste of this.</p>
<p>Best regards</p>
<p>Thomas</p>