Specifying GPU to use with DataParallel

username: hyperdo
cooked: <p>I have a <code>DataParallel</code> model that has been sent to the GPUs via <code>.to('cuda')</code>. I also have some processes calling this model in parallel at various points. It seems like because these are forward passes of batch size 1, they are automatically allocated to CUDA:0, which results in disproportionately high GPU utilization on that device.</p>
<p>How do I specify which GPU is used in a forward pass? I don’t want to have to do any sending of parameters / state dicts. Thanks.</p>

username: InnovArul
cooked: <p>I do not think there is any option/parameter to tell <code>DataParallel</code> which GPUs to use for inference/forward.  If you are only doing inference, won’t it be easy to maintain models in each GPU manually (<code>mode.to(device)</code>) instead of <code>DataParallel</code>?</p>

username: hyperdo
cooked: <p>Yes, I’m only doing inference. I thought that if we had <code>model_new=model.to('cuda:1')</code>, then after an update to <code>model</code>, the parameters wouldn’t be synced. Is that not right? Thanks.</p>

username: InnovArul
cooked: <p>If you are only doing inference in <code>.eval()</code> (not <code>.train()</code> mode), there is no need for parameter sync. Isn’t it?</p>

username: hyperdo
cooked: <p>I thought that if you made a model <code>model_new = model.to('cuda:1')</code>, and then updated the parameters of <code>model</code> with <code>model_optimizer.step()</code>, then the parameters of <code>model_new</code> would be out of sync / differ from <code>model</code>?</p>

username: alex.veuthey
cooked: <p>According to <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">the tutorial</a>, DataParallel splits “data” automatically across available GPUs. I’m pretty sure it only works on batches, so you need batches of more than 1 sample, otherwise it might (a) make no sense to split data, (b) be very inefficient due to synchronization…</p>
<p>Do you even have any usage on other GPUs than the first one? If you have batch sizes of 1, nothing would be split across GPUs and only CUDA:0 would be used.</p>

username: pietern
cooked: <p>Indeed, batch size 1 with <code>DataParallel</code> goes to first specified device (or defaults to <code>cuda:0</code>).</p>
<p>If you want to do inference with batch size 1 there is no need to use <code>nn.DataParallel</code>. This would be useful only if you have a much larger batch that you want to automatically split and automatically run on multiple GPUs. If you want to manually balance batches of size 1 you’re going to have to copy the model yourself and round robin over it. You’re right that the weights are not automatically updated if the source model is updated, because they are different tensors at that point. You’ll have to re-initialize the per-device models every time after running an optimizer on a single source model. In fact, this is exactly how <code>nn.DataParallel</code> works under the covers. On every call to <code>forward</code> it replicates a single module over N devices, scatters the input, runs <code>forward</code> on every one of them, and gathers the outputs. This repeats for every iteration.</p>