How to use torch.nn.functional.conv2d with multi gpu?

username: Zhang_Chi
cooked: <p>How to use torch.nn.functional.conv2d for computing with multi gpus?</p>

username: Zhang_Chi
cooked: <p>anyone knows about it?</p>

username: ptrblck
cooked: <p>You won’t be able to use this API on different GPUs directly and could split the input in e.g. the batch dimension, send each chunk to a GPU, and apply the convolution there.</p>

username: Zhang_Chi
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="113954">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>a GPU, and apply the convolution there.</p>
</blockquote>
</aside>
<p>thanks for the reply. after sending each chunk to different gpus, how to apply the convolution in parallel?</p>

username: ptrblck
cooked: <p>You could directly call them:</p>
<pre><code class="lang-python">out1 = F.conv2d(in1, weight1)
out2 = F.conv2d(in2, weight2)
</code></pre>
<p>where <code>inX</code> and <code>weightX</code> is on the <code>cuda:x</code> device.</p>

username: Zhang_Chi
cooked: <p>but  these two operations are done sequentially, right?  how to make it parallel?</p>

username: ptrblck
cooked: <p>The kernel scheduling would be executed sequentially by the CPU and thus the launches might be delayed, but the kernel execution on the GPU would be performed in parallel.<br>
You would see this effect by profiling a sufficiently large workload using e.g. Nsight Systems.</p>

username: Zhang_Chi
cooked: <p>thanks for the explanation.<br>
so  out2 = F.conv2d(in2, weight2) will start before F.conv2d(in1, weight1) is finished on GPU 1, right?</p>
<p>BTW, what if the weights are shared in two lines, should I do it by:<br>
out1 = F.conv2d(in1, weight.cuda(0))<br>
out2 = F.conv2d(in2, weight.cuda(1))</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Zhang_Chi" data-post="8" data-topic="113954">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/48/1582_2.png" class="avatar"> Zhang_Chi:</div>
<blockquote>
<p>so out2 = F.conv2d(in2, weight2) will start before F.conv2d(in1, weight1) is finished on GPU 1, right?</p>
</blockquote>
</aside>
<p>If the CPU is fast enough to schedule it and the actual kernel execution takes more time than the launch, then yes.<br>
You won’t be able to see any overlap with a tiny workload, as the kernel launch overheads would be larger than the actual GPU workload, i.e. <code>kernel1</code> finishes before the CPU can schedule the launch of <code>kernel2</code>.</p>
<aside class="quote no-group" data-username="Zhang_Chi" data-post="8" data-topic="113954">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/48/1582_2.png" class="avatar"> Zhang_Chi:</div>
<blockquote>
<p>BTW, what if the weights are shared in two lines, should I do it by:</p>
</blockquote>
</aside>
<p>Yes, you have to move the parameters to the appropriate device before executing the operation.</p>

username: Zhang_Chi
cooked: <p>Thank you very much for the reply!</p>