C++ Custom Operator is slow in this case

Sami_Hassan: <pre><code class="lang-auto">#include &lt;torch/extension.h&gt;

#include &lt;iostream&gt;
#include &lt;vector&gt;

std::vector&lt;at::Tensor&gt; lltm_forward(torch::Tensor dummy, 
torch::Tensor mult1, int element_per_filter, int mult1_shape_0, 
int y_index_2, int mult1_shape_1)
{
    // auto dummy1 = dummy.accessor&lt;float, 2&gt;();
    // auto mult11 = mult1.accessor&lt;float, 2&gt;();
    int iterator = 0;
    float number = 0;
    //mult1 = mult1.data&lt;float&gt;;
    int temp = 0;
    int filternumber = -1;
    for (int i = 0; i &lt; mult1_shape_0 + 1; i = i + y_index_2)
    {
        for (int j = 0; j &lt; mult1_shape_1; j++)
        {
            number = 0;

            for (int k = iterator; k &lt; i; k++)
            {
                number = number + mult1[k][j].item().to&lt;double&gt;();
                temp = k;
            }
            dummy[temp][j] = number;


    return {dummy};
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)
{
    m.def("nice", &amp;lltm_forward, "LLTM forward");
}

</code></pre>
<p>This code works but it around 400 x slower than it’s python variant. When i replace  mult1[k][j].item().to(); with a float number (let say i from the loop) it becomes 66%  faster which is what i want from this operator. I was thinking if a float tensor is passed at the python end instead of torch::Tensor mult1 and i loop through it it would be faster (by just doing mult1[k][j]), i have tried to convert mult1 to float as follows but couldn’t make that work either.</p>
<pre><code class="lang-auto">    auto x = mult1.to(torch::kFloat32);
</code></pre>
<p>Which still gives:</p>
<pre><code class="lang-auto">: error: cannot convert ‘at::Tensor’ to ‘float’ in assignment
                 number = number + x[k][j]; //.item().to&lt;double&gt;();
</code></pre>
<p>Guidance on this issue is required from respected members. Is it possible to send numpy arrays to C++ operator perform looping on it and return a numpy array. I have tried that too but could not make to work yet.</p>

albanD: <p>Hi,</p>
<p>Is the input Tensor on GPU by any chance?<br>
Also what is the python counterpart that is faster?</p>

Sami_Hassan: <p>Thank you for your reply. Actually Tensor were converted to numpy for python looping, but i was thinking c++ loops are much better so a custom operator is a good choice. Python loops run on numpy arrays. Here is a snippet to demonstrate both parts</p>
<pre><code class="lang-auto">    import time

    #######################
    # Calling C++ made operator here
    a = torch.from_numpy(dummy).to(torch.float)
    b = torch.from_numpy(mult1).to(torch.float)
    start = time.time()  # Torch passed to c++ operator a and b
    nani = nice.nice(
        a, b, element_per_filter, mult1.shape[0], yindex_2, mult1.shape[1])
    #######################
    end = time.time()
    print(end - start, "C++")

    start = time.time()

    for i in range(0, mult1.shape[0]+1, yindex_2):   # python looping on numpy arrays

        for j in range(0, mult1.shape[1], 1):                
            number = 0
            for k in range(iterator,   i,  1):            
                number = number +mult1[k][j]          
                temp = k
            dummy[temp][j] = number
    end = time.time()
    print(end - start, "Python Loop")

</code></pre>

Shisho_Sama: <p>I had similar experience with libtorch. Looking at your code, you are not using any libtorch specific functionality that binds to using tensors only. So you should be able to easily swap tensor with a vector.<br>
Just converting the <code>mult1</code> from a tensor into a vector, would greatly boost your performance.</p>

Sami_Hassan: <p>If it’s possible can you please provide a snippet wrt to this code. I am not using any tensor related functionality that’s why i want to move these loops to c++</p>

Matej_Kompanek: <p>I tried rewriting it with raw arrays (without really knowing what sizes of tensors you are using), seems much faster</p>
<pre><code class="lang-auto">std::vector&lt;at::Tensor&gt; lltm_forward2(torch::Tensor dummy, torch::Tensor mult1, int element_per_filter, int mult1_shape_0, int y_index_2, int mult1_shape_1)
{
	TORCH_CHECK(dummy.is_contiguous(), " dummy must be a continous tensor");
	TORCH_CHECK(mult1.is_contiguous(), " mult1 must be a continous tensor");
	TORCH_CHECK(!dummy.is_cuda(), " dummy can't be CUDA tensor");
	TORCH_CHECK(!mult1.is_cuda(), " mult1 can't be CUDA tensor");

	auto dummyPtr = dummy.data_ptr&lt;float&gt;();
	auto mult1Ptr = mult1.data_ptr&lt;float&gt;();

	const size_t dummyPtr_stride = dummy.size(1);
	const size_t mult1Ptr_stride = mult1.size(1);

	int iterator = 0;
	float number = 0.0f;

	int temp = 0;
	int filternumber = -1;

	for (int i = 0; i &lt; mult1_shape_0 + 1; i = i + y_index_2)
	{
		for (int j = 0; j &lt; mult1_shape_1; j++)
		{
			number = 0.0f;

			for (int k = iterator; k &lt; i; k++)
			{
				number += mult1Ptr[k * mult1Ptr_stride + j];
				temp = k;
			}

			dummyPtr[temp * dummyPtr_stride + j] = number;
		}
	}

	return { dummy };
}
</code></pre>
<p>Tested it with this:</p>
<pre><code class="lang-auto">void main(int argc, char** argv)
{
	try
	{
		int SZ1 = 30;
		int SZ2 = 40;

		torch::Tensor dummySrc = torch::randn({ SZ1, SZ2 });
		torch::Tensor mult1Src = torch::randn({ SZ1, SZ2 });

		torch::Tensor result1, result2;

		{
			auto dummy = dummySrc.detach().clone();
			auto mult1 = mult1Src.detach().clone();
			std::chrono::steady_clock::time_point begin1 = std::chrono::steady_clock::now();
			auto output1 = lltm_forward(dummy, mult1, 0, mult1.size(0), 1, mult1.size(1));
			std::chrono::steady_clock::time_point end1 = std::chrono::steady_clock::now();
			std::cout &lt;&lt; "Time difference = " &lt;&lt; std::chrono::duration_cast&lt;std::chrono::microseconds&gt;(end1 - begin1).count() &lt;&lt; "[µs]" &lt;&lt; std::endl;
			result1 = output1[0];

			//std::cout &lt;&lt; output1[0] &lt;&lt; std::endl;
		}

		{
			auto dummy = dummySrc.detach().clone();
			auto mult1 = mult1Src.detach().clone();
			std::chrono::steady_clock::time_point begin2 = std::chrono::steady_clock::now();
			auto output2 = lltm_forward2(dummy, mult1, 0, mult1.size(0), 1, mult1.size(1));
			std::chrono::steady_clock::time_point end2 = std::chrono::steady_clock::now();
			std::cout &lt;&lt; "Time difference = " &lt;&lt; std::chrono::duration_cast&lt;std::chrono::microseconds&gt;(end2 - begin2).count() &lt;&lt; "[µs]" &lt;&lt; std::endl;

			result2 = output2[0];

			//std::cout &lt;&lt; output2[0] &lt;&lt; std::endl;
		}

		std::cout &lt;&lt; "Max error: " &lt;&lt; (result2 - result1).abs().max() &lt;&lt; std::endl;

		system("PAUSE"); return;
	}
	catch (std::runtime_error&amp; e)
	{
		std::cout &lt;&lt; e.what() &lt;&lt; std::endl;
	}
	catch (const c10::Error&amp; e)
	{
		std::cout &lt;&lt; e.msg() &lt;&lt; std::endl;
	}

	system("PAUSE");
}
</code></pre>
<p>EDIT: fixed stride values<br>
EDIT2: changes from C++ assert to TORCH_CHECK</p>

Sami_Hassan: <p>Thank you very much. This is exactly what i needed. Just one more question how to adjust the stride for</p>
<pre><code class="lang-auto">`number += mult1Ptr[k * mult1Ptr_stride + j];         // mult1[k][j]

dummyPtr[temp * dummyPtr_stride + j] = number;`   // dummy[temp][j] = number
</code></pre>
<p>As i don’t get the same answer for it as compared to the original loop.<br>
(I am not very efficient in using pointers)</p>

Matej_Kompanek: <p>Sorry, it should have been</p>
<pre><code class="lang-auto">const size_t dummyPtr_stride = dummy.size(1);
const size_t mult1Ptr_stride = mult1.size(1);
</code></pre>
<p>I was using same size for both dimensions so it worked.<br>
I also changed it in the previous post</p>

Sami_Hassan: <p>You are awesome. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thank you it is</p>
<pre><code class="lang-auto">0.0011734962463378906 C++
0.8707504272460938 Python Loop
</code></pre>
<p>Edit: Entered Values of speedup.</p>