RuntimeError: no support for _allgather_base in Gloo process group

Siladittya_Manna: <p>I am trying to run DINOv2 on WIndows. So far, I have been able to run it on windows by using “Gloo” backend for <code>torch.distributed</code> and hasn’t faced any issues except the error mentioned in the title.</p>
<p>I followed the change suggested here in <a href="https://github.com/pytorch/pytorch/issues/74041" class="inline-onebox" rel="noopener nofollow ugc">FSDP does not work on GLOO backend · Issue #74041 · pytorch/pytorch · GitHub</a>, wherein I changed the</p>
<pre><code class="lang-auto">    if group is None:
        default_pg = _get_default_group()
        work = default_pg._allgather_base(output_tensor, input_tensor)
    else:
        work = group._allgather_base(output_tensor, input_tensor)
</code></pre>
<p>in <code>torch.distributed.distributed_c10d.py</code> (Line no. 2528) in the <code>all_gather_into_tensor</code> function definition</p>
<p>to</p>
<pre><code class="lang-auto">    if group is None:
        default_pg = _get_default_group()
        work = default_pg._allgather_base(output_tensor, input_tensor)
    else:
        if group._get_backend_name() == Backend.GLOO:
            return all_gather([output_tensor], input_tensor, group=group)
        else:
            work = group._allgather_base(output_tensor, input_tensor)
</code></pre>
<p>But it faces another issue which is also mentioned in this issue <a href="https://github.com/pytorch/pytorch/issues/74041" class="inline-onebox" rel="noopener nofollow ugc">FSDP does not work on GLOO backend · Issue #74041 · pytorch/pytorch · GitHub</a>, which is <a href="https://github.com/pytorch/pytorch/issues/74041#issuecomment-1637434611" class="inline-onebox" rel="noopener nofollow ugc">FSDP does not work on GLOO backend · Issue #74041 · pytorch/pytorch · GitHub</a></p>
<p>As per the comments, it seems that after the <strong>all_gather</strong> call the parameters are returned as a list and <strong>_save_to_state_dict</strong> calls a function which expects them as one large tensor. This function appears to be <strong>_local_pre_state_dict_hook</strong> in <strong>torch.distributed.fsdp._state_dict_utils</strong> .</p>
<p>The suggested changes in one comment are as follows (<a href="https://github.com/pytorch/pytorch/issues/74041#issuecomment-1637508197" class="inline-onebox" rel="noopener nofollow ugc">FSDP does not work on GLOO backend · Issue #74041 · pytorch/pytorch · GitHub</a>)</p>
<pre><code class="lang-auto">def _local_pre_state_dict_hook(
    fsdp_state: _FSDPState,
    module: nn.Module,
    *args,
    **kwargs,
) -&gt; None:
    """
    Hook that runs before model.state_dict() is called. Right now, pre-state_dict
    hook is not supported by the PyTorch core. So this API is called from
    `_local_post_state_dict_hook()` to simulate the case.
    """
    #if (
    #    _has_fsdp_params(fsdp_state, module)
    #    and not _module_handles(fsdp_state, module)[0].uses_sharded_strategy
    #):
    #    raise RuntimeError(
    #        "``local_state_dict`` can only be used when parameters are flatten "
    #        "and sharded."
    #    )
    _common_pre_state_dict_hook(module, fsdp_state)
</code></pre>
<p>But, is there any other way to do it, that is, avoid the error with Gloo backend in the <code>all_gather_into_tensor</code> function and also avoid this error when saving the model and the above mentioned change?</p>
<p>The error can be replicated using the code in this repo <a href="https://github.com/sadimanna/dinov2_nosetup_windows" class="inline-onebox" rel="noopener nofollow ugc">GitHub - sadimanna/dinov2_nosetup_windows: PyTorch code and models for the DINOv2 self-supervised learning method in Windows</a> where I have modified the DINOv2 code for running on Windows.</p>
<p>But I guess the issue is associated with the gloo backend, rather than Windows OS.</p>

fegin: <p>One possible way to do state_dict with FSPD without allgather is to use <code>StateDictType.LOCAL_STATE_DICT</code>. However, <code>StateDictType.LOCAL_STATE_DICT</code> is not good for the production use case as the saved state_dict can only be loaded back to the same FSDP model with the same GPUs. Other than this way, all other solutions require the change to FSDP as you mentioned.</p>

Siladittya_Manna: <p><a class="mention" href="/u/fegin">@fegin</a></p>
<p>The <code>FSDPCheckpointer</code> in <code>dinov2.fsdp.__init__.py</code> already contains these two lines</p>
<pre><code class="lang-auto">with FSDP.state_dict_type(self.model, StateDictType.LOCAL_STATE_DICT):
    data["model"] = self.model.state_dict() 
</code></pre>
<p>But the error still occurs.</p>