Doubt about creating a tensor from a gpu pointer

deepfailure: <p>Hello, I have a doubt in regards to the c++ implementation of an already trained model in python. I am new in the use of c++ for pytorch.</p>
<p>Given the trained model, I know how to load it in c++ implementation but I would like to speed up the inference.  I receive the input as a pointer to GPU memory, is there any way to use this pointer as the input to the neural network without doing any extra copy?</p>
<p>I believe if I do:</p>
<p>Given shape1 and shape2 of the tensor, a trained model and a GPU pointer p:</p>
<pre><code class="lang-auto">auto options = torch::TensorOptions().device(torch::kCUDA);
auto tensor_from_cuda_pointer=torch::from_blob(p,{shape1,shape2}, options);
at::Tensor output = module.forward(tensor_from_cuda_pointer).toTensor();
</code></pre>
<p>A tensor from the CUDA memory will be created and the inference will be done. However, how can I be sure that there is no extra copy done here?</p>
<p>Thanks in advance.</p>

deepfailure: <p>To test my issue I’ve tried to create different big sized tensors and measure the gpu memory with the command nvidia-smi:</p>
<ul>
<li>Create tensor1 on gpu and create tensor2 from pointer of tensor1.</li>
<li>Create only tensor1.</li>
<li>Create tensor1 and tensor2 from scratch on gpu</li>
<li>Create tensor1 from scratch on gpu, clone tensor1 and send it to gpu.</li>
</ul>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.
#include "windows.h"
#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
	int mode = 1;
	int shape1 = 20;
	int shape2 = 1600;
	int shape3 = 1600;

	if (mode == 1)
	{	
		// one tensor created from scratch and a second tensor created from tensor1 pointer
		torch::Tensor ones_tensor = torch::ones({shape1, 1, shape2, shape3}, torch::device(torch::kCUDA));
		auto p = ones_tensor.data_ptr&lt;float&gt;();
		auto options = torch::TensorOptions().device(torch::kCUDA);
		torch::Tensor new_tensor_from_pointer = torch::from_blob(p, {shape1, 1, shape2, shape3}, options);
	} else if (mode == 2){
		// one tensor created from scratch to gpu
		torch::Tensor ones_tensor = torch::ones({shape1, 1, shape2, shape3}, torch::device(torch::kCUDA));
	} else if (mode == 3){
		// two tensors created from scratch to gpu
		torch::Tensor ones_tensor = torch::ones({shape1, 1, shape2, shape3}, torch::device(torch::kCUDA));
		torch::Tensor two_tensor = torch::ones({shape1, 1, shape2, shape3}, torch::device(torch::kCUDA));
	}
	else if (mode == 4) 
	{
		// one tensor created from scratch, second tensor created from clone
		torch::Tensor ones_tensor = torch::ones({shape1, 1, shape2, shape3}, torch::device(torch::kCUDA));
		auto new_tensor_cloned = ones_tensor.clone();
		torch::Device device(torch::kCUDA);
		new_tensor_cloned.to(device);
	}
	else
	{
		std::cout &lt;&lt; "Doing nothing";
	}
	Sleep(30000);
}
</code></pre>
<p>The results are the following:</p>
<ul>
<li>Mode 1: 667 MiB</li>
<li>Mode 2: 667 MiB</li>
<li>Mode 3: 863 MiB</li>
<li>Mode 4: 863 MiB</li>
</ul>
<p>So it seems that in fact torch::from_blob does not copy the original tensor which is what I am searching for. However, I was expecting modes 3 and 4 to have approximatedly double the size as Modes 1 and 2, what could be a reason for that?</p>
<p>Thanks in advance.</p>

ptrblck: <aside class="quote no-group" data-username="deepfailure" data-post="2" data-topic="145949">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a3d4f5/48.png" class="avatar"> deepfailure:</div>
<blockquote>
<p>However, I was expecting modes 3 and 4 to have approximatedly double the size as Modes 1 and 2, what could be a reason for that?</p>
</blockquote>
</aside>
<p>The CUDA context would use memory to load kernels, the driver etc.</p>