Why scatter_object_list does not support NCCL?

username: walle_autoscale
cooked: <p>Hi, I was reading the torch.distributed doc, and I found that the doc say <code>scatter_object_list</code> does not support NCCL backend due to tensor based scatter is not supported. But the <code>dist.scatter</code> seems to support NCCL backend. I think these are confilict.</p>
<p>ref: <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.scatter_object_list" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.12 documentation</a></p>
<p>Could anyone explain why?</p>
<p>Thanks a lot!</p>

username: walle_autoscale
cooked: <p>I read the code, and it seems that in <code>torch/distributed/distributed_c10d.py</code>, function <code>scatter_object_list</code>  does not move <code>tensor_list</code> to device before scatter. Why?</p>

username: H-Huang
cooked: <p>Hi, thanks for the raising the issue. Responses to the questions are below.</p>
<aside class="quote no-group" data-username="walle_autoscale" data-post="1" data-topic="160515">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/walle_autoscale/48/52644_2.png" class="avatar"> walle_autoscale:</div>
<blockquote>
<p>Hi, I was reading the torch.distributed doc, and I found that the doc say <code>scatter_object_list</code> does not support NCCL backend due to tensor based scatter is not supported. But the <code>dist.scatter</code> seems to support NCCL backend. I think these are confilict.</p>
<p>ref: <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.scatter_object_list" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.12 documentation </a></p>
<p>Could anyone explain why?</p>
</blockquote>
</aside>
<p>There does seem to be an inconsistency, it looks like NCCL scatter support was introduced recently <a href="https://github.com/pytorch/pytorch/pull/70029" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/70029</a> and the support for scatter_object_list has not been updated. Created an issue on github to track this: <a href="https://github.com/pytorch/pytorch/issues/84571" class="inline-onebox" rel="noopener nofollow ugc">dist.scatter_object_list() NCCL support · Issue #84571 · pytorch/pytorch · GitHub</a></p>
<aside class="quote no-group" data-username="walle_autoscale" data-post="2" data-topic="160515" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/walle_autoscale/48/52644_2.png" class="avatar"> walle_autoscale:</div>
<blockquote>
<p>I read the code, and it seems that in <code>torch/distributed/distributed_c10d.py</code>, function <code>scatter_object_list</code> does not move <code>tensor_list</code> to device before scatter. Why?</p>
</blockquote>
</aside>
<p><code>scatter_object_list()</code> is a distributed collective that is not specific to a certain backend. We cannot assume the device that is used, perhaps the user may have the tensors on CPU and use the <code>gloo</code> backend.</p>

username: kumpera
cooked: <p>This was fixed for 1.12 but I forgot to update the docstring.<br>
The fix for the collective was in: <a href="https://github.com/pytorch/pytorch/pull/79034" class="inline-onebox" rel="noopener nofollow ugc">[distributed] Handle object collectives and NCCL. by kumpera · Pull Request #79034 · pytorch/pytorch · GitHub</a></p>

username: kumpera
cooked: <p>Fixed the docstring in <a href="https://github.com/pytorch/pytorch/pull/84596" class="inline-onebox" rel="noopener nofollow ugc">[c10d] Fix docstring of scatter_object_list by kumpera · Pull Request #84596 · pytorch/pytorch · GitHub</a></p>