New to DataParallel... can I just wrap it around each layer or module?

username: Sam_Lerman
cooked: <p>I have modules composed of modules composed of modules. Some of these modules might not be compatible with DataParallel, I’m not sure, because they don’t return tensors (for instance, they return distributions). I would like to enable parallelization for as much of the architecture as possible… is there any downside to this? Can I just wrap the lowest level layers and modules in <code>nn.DataParallel</code>? As in, can I wrap <code>nn.Sequential</code>s and so on in DataParallel and call it a day? I’m really not sure what the best practice is with this feature.</p>

username: ptrblck
cooked: <p>Yes, you could wrap submodules into <code>nn.DataParallel</code>. Note however that <code>nn.DataParallel</code> is generally slower than <code>DistributedDataParallel</code> (using a single process per device) as it has to e.g. broadcast the modules in each forward pass.</p>

username: Sam_Lerman
cooked: <p>Thanks! So, compared to neither, there wouldn’t be a downside to <code>DataParallel</code>, would there?</p>
<p>And I looked at the <code>DistributedDataParallel</code> docs, but it seems a little more involved than just wrapping submodules… not sure I have the time right now to go that deep.</p>
<p>Also… maybe I’m creating too many branches of questions, but… how does <code>DataParallel</code> work with modules that don’t return tensor outputs, like the one I mentioned that returns a distribution?</p>

username: Sam_Lerman
cooked: <p>And once I’ve wrapped them as follows: <code>nn.DataParallel(module)</code>, is that it? Will it parallelize when multiple GPUs available? How will it know to use GPUs and not CPUs? I on’y want it to parallelize across CUDA devices.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sam_Lerman" data-post="3" data-topic="144344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/48/37165_2.png" class="avatar"> Sam_Lerman:</div>
<blockquote>
<p>how does <code>DataParallel</code> work with modules that don’t return tensor outputs, like the one I mentioned that returns a distribution?</p>
</blockquote>
</aside>
<p>I don’t know and would recommend to check the behavior with a minimal code snippet.<br>
<a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">This blog post</a> describes the internal behavior of <code>nn.DataParallel</code> in more detail and you would see where the inputs are scattered/gathered etc.</p>
<aside class="quote no-group" data-username="Sam_Lerman" data-post="4" data-topic="144344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/48/37165_2.png" class="avatar"> Sam_Lerman:</div>
<blockquote>
<p>Will it parallelize when multiple GPUs available?</p>
</blockquote>
</aside>
<p>Yes, by default all visible devices are used, but you can also specify them in the initialization of <code>nn.DataParallel</code>.</p>

username: Sam_Lerman
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="144344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Yes, by default all visible devices are used, but you can also specify them in the initialization of <code>nn.DataParallel</code>.</p>
</blockquote>
</aside>
<p>Oh, even CPUs? Hmm… it would be convenient if there was a “cuda-only” option</p>

username: ptrblck
cooked: <p>No, all your visible GPUs will be used.</p>

username: Sam_Lerman
cooked: <p>I guess my last question is, does sub-module-wise <code>DataParallel</code> have a cost tradeoff compared to a single <code>DataParallel</code> on the very-outer module?</p>

username: ptrblck
cooked: <p>Yes, I would think so as the scatter/gather ops would be called on each submodule. Check the linked blog post, which describes this, and apply the same logic for each submodule to visualize how this approach would work.</p>

username: Sam_Lerman
cooked: <p>So, with DistributedDataParallel, there would not be this issue? I could distribute submodule-wise, and there should not be this delay?</p>

username: ptrblck
cooked: <p>I don’t think so, since your workflow explicitly pushes modules to different devices and then expects to collect the output on a single device, execute some modules on a single device, and repeat this usage.<br>
If so, then you would also need to write your <code>DDP</code> application in this way.</p>