Best practice for GPU -> CPU -> GPU computation

CharlesLu: <p>Hi everyone,</p>
<p>I’m currently working on a pipeline that involves using OpenCV between two GPU computations. However, I encountered two problems:</p>
<ol>
<li>
<p>OpenCV code runs on the CPU, which means that data must be transferred from the GPU to the CPU and back to the GPU again.</p>
</li>
<li>
<p>The OpenCV function can only process one sample (from a batch) at a time, which slows down the computation.</p>
</li>
</ol>
<p>Here is an example of my code:</p>
<pre><code class="lang-auto"># GPU computation
x = ...  # B, C, H, W

# Pre-allocate memory
outputs = torch.zeros(some_shape, device="cuda")

# Send each sample in the batch to the CPU
for b in range(B):
    curr_x = x[b].cpu().numpy()

    # Use a special sequential algorithm, so no GPU alternatives
    output = cv2.function(...)  

    # send the sample back to the GPU
    output = torch.Tensor(output).cuda()
    outputs[b] = output

# Other GPU computations
...
</code></pre>
<p>The code runs slowly with low CPU and GPU usage.</p>
<p>I would be grateful for any suggestions or insights you may have.<br>
Thank you very much!</p>

ptrblck: <p>I don’t know which <code>function</code> you are calling from OpenCV, but you might want to either check if OpenCV provides the CUDA version of this function or if you could use another library with GPU support (e.g. <code>torchvision</code> using native PyTorch operations or custom kernels).</p>

CharlesLu: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you for your suggestion!</p>
<p>The function I’m using is homography matrix estimation with RANSAC:<br>
<code>cv2.findHomography(..., cv2.RANSAC)</code>.</p>
<p>However, OpenCV does not provide an off-the-shelf CUDA version of it. And existing implementation using PyTorch (e.g., Kornia) is even slower than OpenCV.</p>
<p>Moreover, both implementations require a single sample as input, instead of a batch.</p>
<p>So I was wondering if it is possible to optimize the code from the perspective of data transferring or parallelization. Do you have any suggestions on this?</p>
<p>Thanks again for your help!</p>

ptrblck: <aside class="quote no-group" data-username="CharlesLu" data-post="3" data-topic="177608">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charleslu/48/59531_2.png" class="avatar"> CharlesLu:</div>
<blockquote>
<p>And existing implementation using PyTorch (e.g., Kornia) is even slower than OpenCV.</p>
</blockquote>
</aside>
<p>I haven’t profiled these implementations and don’t know what inputs you are using, but note that even if OpenCV might be faster in isolation on the CPU, you will still pay the penalty of moving the data between the device and host and thus synchronizing the code.<br>
Assuming <code>kornia</code> provides a GPU implementation no syncs would be added and the end2end time might still be faster. But as I said, I did not profile any of these methods.</p>
<aside class="quote no-group" data-username="CharlesLu" data-post="3" data-topic="177608">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charleslu/48/59531_2.png" class="avatar"> CharlesLu:</div>
<blockquote>
<p>Moreover, both implementations require a single sample as input, instead of a batch.</p>
</blockquote>
</aside>
<p><a href="https://kornia.readthedocs.io/en/latest/geometry.homography.html"><code>kornia.geometry.homography.find_homography_dlt</code></a> seems to expect batched inputs based on the docs. CC <a class="mention" href="/u/edgarriba">@edgarriba</a> to correct me.</p>
<aside class="quote no-group" data-username="CharlesLu" data-post="3" data-topic="177608">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charleslu/48/59531_2.png" class="avatar"> CharlesLu:</div>
<blockquote>
<p>So I was wondering if it is possible to optimize the code from the perspective of data transferring or parallelization.</p>
</blockquote>
</aside>
<p>I’m unsure how you want to do it. You could try to move the data with <code>non_blocking=True</code>, but if the next operation is actually needing the transferred tensor you won’t be able to overlap the transfer.</p>

CharlesLu: <p>Thank you for the prompt reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="177608">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I haven’t profiled these implementations and don’t know what inputs you are using, but note that even if OpenCV might be faster in isolation on the CPU, you will still pay the penalty of moving the data between the device and host and thus synchronizing the code.<br>
Assuming <code>kornia</code> provides a GPU implementation no syncs would be added and the end2end time might still be faster. But as I said, I did not profile any of these methods.</p>
</blockquote>
</aside>
<p>Yes, I actually profiled the runtime of <code>kornia.geometry.ransac</code> (which is equivalent to <code>cv2.findHomography(..., cv2.RANSAC)</code>. Unfortunately, even with the overhead of data transferring, the OpenCV version still seems to be about 10 times faster than the Kornia implementation.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="177608">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><a href="https://kornia.readthedocs.io/en/latest/geometry.homography.html" rel="noopener nofollow ugc"><code>kornia.geometry.homography.find_homography_dlt</code></a> seems to expect batched inputs based on the docs. CC <a class="mention" href="/u/edgarriba">@edgarriba</a> to correct me.</p>
</blockquote>
</aside>
<p>Thanks for bringing up this function. Yes, I’ve also tested this one. The computation is a lot faster, but the lack of robust estimation (like RANSAC) leads to worse results in my case.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="177608">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m unsure how you want to do it. You could try to move the data with <code>non_blocking=True</code>, but if the next operation is actually needing the transferred tensor you won’t be able to overlap the transfer.</p>
</blockquote>
</aside>
<p>Lastly, thanks for this suggestion! Unfortunately, as you pointed out, my next operation actually needs the transferred tensor.</p>
<p>I was thinking about:</p>
<ol>
<li>
<p>Whether sending the entire <code>x</code> batch to the CPU leads to any speedup over sending each sample in <code>x</code> separately. (The speedup will probably be marginal, though)</p>
</li>
<li>
<p>Whether it is possible to run the OpenCV function on all samples in the batch concurrently with multi-processing.</p>
</li>
</ol>
<p>But in general, I guess it is difficult to significantly improve the runtime without large edits to the pipeline or to the OpenCV/Kornia source code.</p>
<p>Thank you again for your time and help:)</p>

ptrblck: <ol>
<li>
<p>Yes, I would expect to see a higher bandwidth if the transfer can be batched instead of calling separate copy kernels on each sample.</p>
</li>
<li>
<p>I don’t know if that’s possible.</p>
</li>
</ol>