Cuda tensor to cpu is very time-consuming

ismymajia: <p>cpu tensor to cuda tensor is very fast. eg: 500x300 image from cpu to cuda tensor cost 2ms.<br>
but 500x300 image from cuda to cpu tensor cost 100ms.</p>
<p>that is :<br>
auto cuda_tensor = torch::autograd::make_variable(batchInputs, false).to(torch::kCUDA); //2ms<br>
auto output = module-&gt;forward(cuda_tensor).toTensor(); //18ms<br>
auto output1 = torch::squeeze(output); //0ms<br>
auto output2 = torch::exp(output1); //0ms<br>
auto image_cpu = output2.cpu(); // 100ms</p>
<p>output2.cpu() cost 100ms,  how can this be optimized?</p>

albanD: <p>Hi,</p>
<p>Keep in mind that the cuda api is asynchronous and only synchronize if you ask for it explicitly with <code>torch.cuda.synchronize()</code> or if you read some value back to the cpu (by calling <code>.cpu()</code> for example).<br>
So the timing you measure here for the line that contains <code>.cpu()</code> is actually the time of doing all computations + sending the tensor back.</p>

Si_laopi: <p>Hi, I have the same problem here. But how to use torch.cuda.synchronize() in C++?</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">CUDAStream stream = getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>

Si_laopi: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="34095">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This should work:</p>
</blockquote>
</aside>
<p>Thank you very much, it works for me. But I have another question. After I add cudaStreamSynchronize in my code, the time consumption of tensor.cpu() does reduced. However, I found out that cudaStreamSynchronize  itself cost lots of time, like 70ms which is almost the same time tensor.cpu() cost before I add cudaStreamSynchronize to the code. So, whether I use cudaStreamSynchronize  or not, this extra time consumption is inevitable, is that right? If it is right, is there any way to reduce such time? Like reduce image number of the input tensor?</p>

ptrblck: <p>This is expected, since the <code>cpu()</code> call is blocking and waiting for all asynchronous CUDA operations to finish. If you don’t synchronize manually, the <code>cpu()</code> call will add the synchronization and thus accumulate the CUDA kernel runtime.<br>
Since you are now synchronizing manually, the runtime is spent in the kernel and the <code>cpu()</code> call executes just the device2host copy.</p>
<aside class="quote no-group" data-username="Si_laopi" data-post="5" data-topic="34095">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/si_laopi/48/22903_2.png" class="avatar"> Si_laopi:</div>
<blockquote>
<p>is there any way to reduce such time? Like reduce image number of the input tensor?</p>
</blockquote>
</aside>
<p>Yes, you could reduce the batch size or the spatial size of the input images.<br>
However, note that while reducing the workload the CUDA operations might reduce the wall-clock time, but the overall performance might be reduces, if you calculate the time/inputs.</p>

Si_laopi: <p>I understand. Thank you very much.</p>