Multi-node multi-GPU training, partly NVlink

username: AlbertZeyer
cooked: <p>I want to run some multi-node multi-GPU training where some GPUs are connected via NVlink but potentially/probably not all of them (but I don’t really know in advance).</p>
<p>How would I ideally do that with PyTorch?</p>
<p>For the reduce, I ideally would want that it does it in the most efficient way possible, i.e. first reduce over the NVlink connected subsets as far as possible, and then over network, and then broadcasted again over NVlink, or sth like that.</p>
<p>Using NCCL probably does not work? So Gloo then? Or both together, and doing it all in a more custom way?</p>
<p>Doing average on gradients is probably also not the most efficient, so I was thinking to do model parameter average instead. (Just like <a href="https://returnn.readthedocs.io/en/latest/advanced/multi_gpu.html" rel="noopener nofollow ugc">this</a>.)</p>

username: ptrblck
cooked: <p>NCCL should detect all NVLinks and use these if possible. You might want to remap the devices via <code>CUDA_VISIBLE_DEVICES</code> if needed to make sure your desired communication operations are using NVLink for heavy comm ops.</p>

username: AlbertZeyer
cooked: <p>There are sometimes cases where CPU ↔ ethernet ↔ CPU reduction would still be faster. E.g. we do epoch-based training, and the data loaders on the workers produce not necessarily the same amount of batches. So, at the beginning of every step, we check if every worker has still data (otherwise we finish the epoch). This is the code:</p>
<pre data-code-wrap="python"><code class="lang-python">_has_data = torch.tensor([extern_data_raw is not None], dtype=torch.int8)
if self._torch_distributed_ctx:
    # use all reduce to check if all workers have data, if at least one worker does not have data,
    # all workers finish this epoch
    torch.distributed.all_reduce(_has_data, op=torch.distributed.ReduceOp.MIN)
if not _has_data[0]:
    break
</code></pre>
<p>Originally we used NCCL here, and moved the tensor to GPU, and then back to CPU. This was much slower than keeping this code on CPU, and using <code>dist.init_process_group(backend=None)</code> to init both NCCL and Gloo.</p>
<p>This makes sense to me. Or do you think this is unexpected and using NCCL should always just be as fast? If yes, then what could be the problem? If no, this means, we should better use Gloo and leave tensors on CPU for those cases?</p>

username: ptrblck
cooked: <p>I don’t fully understand why you are moving data from the GPU to the CPU in the first place as it could easily cause the bottleneck. Due to this NCCL might not underperform but your data movement.</p>

username: AlbertZeyer
cooked: <p>In the example with <code>_has_data</code>, this is on CPU. And here, I guess Gloo would make more sense, right?</p>
<p>But for all data which is anyway on GPU, I can (should) use NCCL, no matter if NVlink is available or not?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="AlbertZeyer" data-post="5" data-topic="193179">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/albertzeyer/48/45362_2.png" class="avatar"> AlbertZeyer:</div>
<blockquote>
<p>In the example with <code>_has_data</code>, this is on CPU. And here, I guess Gloo would make more sense, right?</p>
</blockquote>
</aside>
<p>I don’t know as I’m not using gloo and usually don’t want to hold data on the CPU.</p>
<aside class="quote no-group" data-username="AlbertZeyer" data-post="5" data-topic="193179">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/albertzeyer/48/45362_2.png" class="avatar"> AlbertZeyer:</div>
<blockquote>
<p>But for all data which is anyway on GPU, I can (should) use NCCL, no matter if NVlink is available or not?</p>
</blockquote>
</aside>
<p>Yes, NCCL should give you the best performance for data stored on the GPU.</p>