Libtorch low latency problem

Iwein: <p>I use std::memcpy to copy an image, given by a high speed camera, to a buffer. The overal idea is that I copy &amp; buffer camera scanlines into a std::vector; when the capturing is finished I’ll construct a libtorch tensor from this. This tensor is then used further in the software to do image procesing.</p>
<p>This works fine: the time needed to copy the contents to the buffer is typically ~1 millisecond;</p>
<pre><code class="lang-auto">torch::from_blob((char*)image-&gt;buf, { res.spatial, res.spectral }, torch::TensorOptions().dtype(torch::kUInt8)).unsqueeze(0).clone()
</code></pre>
<p>…and the result is correct in terms of memory layout.</p>
<p>However: sometimes the copy time is 40-80 milliseconds, which is 40-80 times (!!) more than the expected time.</p>
<ol>
<li>Why is this? and how do I solve this?</li>
<li>Does anyone notice some code aspect that can be improved (for speed)?</li>
</ol>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; result;          
result.reserve(nbrOfLines);

camera-&gt;startCameraStream();
auto tensor = torch::empty({res.spatial, res.spectral }, torch::kUInt8);
 for (auto lineIndex = 0; lineIndex &lt; nbrOfLines; lineIndex++)
 {
    struct image_buffer* image;
    auto success = camera-&gt;getImage(&amp;image); 
    while (!success)                
        success = camera-&gt;getImage(&amp;image);             

    result.push_back(torch::from_blob((char*)image-&gt;buf, { res.spatial, res.spectral }, torch::TensorOptions().dtype(torch::kUInt8)).unsqueeze(0).clone());

    // Return the result so it can be filled again...
    camera-&gt;returnImage(&amp;image);
 }

 auto framegrab = torch::cat(result,0);
</code></pre>

smth: <p>I’d suggest you try two things:</p>
<ol>
<li>
<p>Run the code with the no_grad guard on, so that there’s even lesser overhead overall: <a href="https://pytorch.org/cppdocs/api/typedef_namespacetorch_1abf2c764801b507b6a105664a2406a410.html#typedef-torch-nogradguard" class="inline-onebox">Typedef torch::NoGradGuard — PyTorch main documentation</a></p>
</li>
<li>
<p>Run the code with <code>torch::set_num_threads(1);</code> just in case multithreading is somehow causing some kind of cache trash.</p>
</li>
</ol>
<p>The other common reason that you’d see a big stall like that is because your memory allocator needs to defragment and realloc. That’s more of an “above pytorch” problem, but you can switch your programs allocator to jemalloc which is often faster.</p>

Iwein: <p>Thanks for the advice <a class="mention" href="/u/smth">@smth</a>!</p>
<p>Good tip: I’ll implement the <code>NoGradGuard</code> and <code>torch::set_num_threads(1);</code> tomorrow.  There was also an interesting suggestion from a college of mine to do the following:</p>
<p>Preallocate a huge tensor ‘torch::tensor framegrab = torch::zeros({<span class="hashtag-raw">#lines</span>, <span class="hashtag-raw">#colums</span>, <span class="hashtag-raw">#wavelengths</span>}’ before the actual grabbing (for each line index…), and then fill it up with scanlines.<br>
Something in the order of:</p>
<p><code>memcpy(framegrab[lineindex, :, :].data(), image_buffer)</code></p>
<p>The problem here is that I don’t know how to copy the 2D scanlines (cols x wavelengths) into the 3D tensor by using e.g. memcpy.</p>
<p>I also think (acctually i’m quite sure) that you rooted the cause of the problem to the memory allocator, but I have no clue on how to set the allocator to jemalloc. The code runs on Windows 10 with a visual studio compiler, C++17, libtorch.</p>

smth: <p>you can actually do what your friend suggested much more efficiently than memcpy by using <code>copy_</code>. And yes, it’s probably better than a result vector + torch::cat.</p>
<p>Something like:</p>
<pre><code class="lang-auto">torch::Tensor result_tensor = torch::from_blob((char*)image-&gt;buf, { res.spatial, res.spectral }, torch::TensorOptions().dtype(torch::kUInt8)).unsqueeze(0);
framegrab[lineindex, :, :].copy_(result_tensor)
</code></pre>
<p>It’s usually faster than memcpy because we use SIMD and multiple threads to do the copy in parallel (memcpy by default usually restricts to one thread).</p>