Getting a eError: Default process group has not been initialized, please make sure to call init_process_group

Jordan_Howell: <p>Hello,</p>
<p>I’ve been trying to move a model from a single GPU to a machine I’ve rented with four GPUs.  I used the <code>DistributedDataParralel</code> command and I’m getting the following error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-30-d1064068ed08&gt; in &lt;module&gt;
      6 exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
      7 combined_model = combined_model.cuda()
----&gt; 8 combined_model = DDP(combined_model)

~/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py in __init__(self, module, device_ids, output_device, dim, broadcast_buffers, process_group, bucket_cap_mb, find_unused_parameters, check_reduction)
    271 
    272         if process_group is None:
--&gt; 273             self.process_group = _get_default_group()
    274         else:
    275             self.process_group = process_group

~/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py in _get_default_group()
    266     """
    267     if not is_initialized():
--&gt; 268         raise RuntimeError("Default process group has not been initialized, "
    269                            "please make sure to call init_process_group.")
    270     return _default_pg

RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
</code></pre>
<p>Here is the code I’m using to try and parallelize the model:</p>
<pre><code class="lang-auto">from torch.nn.parallel import DistributedDataParallel as DDP
torch.manual_seed(101)
combined_model = Image_Embedd(embedding_size=train_categorical_embedding_sizes)
criterion = torch.nn.NLLLoss().cuda()
optimizer = torch.optim.Adam(combined_model.parameters(), lr=0.001)
scheduler = ReduceLROnPlateau(optimizer, 'min', patience = 4, verbose = True, min_lr = .00000001)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
combined_model = combined_model.cuda()
combined_model = DDP(combined_model)
</code></pre>
<p>Has anyone ran into this error before?  I’m on a ubuntu environment with jupyter notebook.</p>

ptrblck: <p>You would have to setup your environment as described <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#basic-use-case">here</a>.</p>

mrshenli: <p><a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">This</a> is a minimum example.</p>

Hamzeh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have trained my model with DistributedDataParallel and have saved the model like this: torch.save(model.module.state_dict(), ‘TS_Model.pth’).<br>
I want to load my model in another machine that has not GPU device, but I get various errors.</p>
<p>1- When I load the model like this:<br>
model = Network()<br>
state_dict = torch.load('TS_model.pth, map_location=torch.device(‘cpu’)).module.state_dict()<br>
model.load_state_dict(state_dict)<br>
model.eval()<br>
I get this error: “Default process group has not been initialized, please make sure to call init_process_group.”</p>
<p>2- When I initialize the environment just like training process and then load the model, I get this error: “Distributed package doesn’t have NCCL built in”</p>
<p>I can run this code on my machine totally fine, but I cannot load it in another machine.<br>
How can I overcome this issue?</p>

ptrblck: <p>The code snippets do not quite match.<br>
Assuming you’ve stored the <code>state_dict</code> alone (as would be the recommended way) via:</p>
<pre><code class="lang-python">torch.save(model.module.state_dict(), 'TS_Model.pth')
</code></pre>
<p>then <code>torch.load('TS_Model.pth')</code> should only return the <code>state_dict</code> (mapped to the CPU).<br>
However, it seems you are trying to load the “entire” model via:</p>
<pre><code class="lang-python">state_dict = torch.load('TS_model.pth', map_location=torch.device('cpu')).module.state_dict()
</code></pre>
<p>as you are calling <code>.module.state_dict()</code> on the output of <code>torch.load</code>.<br>
This would mean that <code>TS_model.pth</code> (note the lowercase <code>m</code> in <code>model</code>) would contain the entire DDP model stored as <code>torch.save(model, path)</code> and would thus try to reinitialize it with the DDP setup and will fail on the other machine.</p>

Hamzeh: <p>When I try<br>
<code>state_dict = torch.load('TS_model.pth', map_location=torch.device('cpu'))</code><br>
I get this error : "<strong>Missing key(s) in state_dict: “branch1.backbone.conv1.0.weight”,“branch1.backbone.conv1.1.weight”</strong> and so on to the end of the model structure (i.e branch1.backbone.conv2 and … ).</p>

ptrblck: <p>This would mean that the keys of the stored <code>state_dict</code> and the current <code>state_dict</code> of the model do not match, which is caused by a change in the model architecture (e.g. if you are re-wrapping the model into another parent model etc.).<br>
Check which keys are mismatches and where the additional or missing attributes are coming from.</p>

Hamzeh: <p>Eventually I loaded the model using:</p>
<pre><code class="lang-auto">state_dict = torch.load('TSmodel.pth',map_location='cpu')
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
        name = k[7:]  # remove `module.`
        new_state_dict[name] = v
    # load params
    model.load_state_dict(new_state_dict)
    model.eval()
</code></pre>
<p>But when I feed the input to the model I receive this error:</p>
<pre><code class="lang-auto">output = model(img[None, :], step=1)
</code></pre>
<p><strong>NVTX functions not installed. Are you sure you have a CUDA build?</strong></p>

ptrblck: <p>Did you add nvtx markers to to forward pass somewhere? If so, you would need to remove them if you want to use a pure CPU build for the model execution.</p>

Hamzeh: <p>Honestly I do not know whether these markers have been added or not. How can I remove them from the environment? (Just to remind I want to load my model in another machine that has not GPU device)<br>
BTW I have torch 1.13.0.dev20220614+cpu on this machine.</p>

ptrblck: <p>I don’t see how the <code>state_dict</code> loading would add them so are you even able to run the plain CPU model (without trying to load the pretrained <code>state_dict</code>) on the CPU using some random input?<br>
If not, where does this model come from as PyTorch would not automatically add GPU-specific code to your model.</p>

Hamzeh: <p>So I tried random input, but I have the same error.<br>
Do I have to change the model saving code?</p>

Hamzeh: <p>When I want to define my model I have to use SynchBatchNorm for norm_layer</p>
<pre><code class="lang-auto">model = Network(num_classes=11, criterion=nn.CrossEntropyLoss, pretrained_model=None, norm_layer=SyncBatchNorm)
</code></pre>
<p>which SyncBatchNorm uses cuda</p>

ptrblck: <p>You might want to replace it with a plain <code>nn.BatchNormXd</code> layer as <code>SyncBatchNorm</code> expects to run in a <code>DDP</code> setup with a single process per GPU.</p>

Hamzeh: <p>Yes, eventually worked <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20">!!</p>
<p>Many thanks for your help and time <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20">.</p>

winchest: <p>HI! I have the same problem but I don’t know why. I didn’t calling <code>.module.state_dict()</code> on the output of <code>torch.load</code> but the same error still happened.<br>
The save code:</p>
<pre><code class="lang-auto">if (opts.local_rank == 0):
        if (opts.checkpoint_epochs != 0 and epoch % opts.checkpoint_epochs == 0) or epoch == opts.n_epochs - 1:
            print('Saving model and state...')
            torch.save(
                {
                    'model': get_inner_model(model).state_dict(),
                    'optimizer': optimizer.state_dict(),
                    'rng_state': torch.get_rng_state(),
                    'cuda_rng_state': torch.cuda.get_rng_state_all(),
                    'baseline': baseline.state_dict()
                },
                os.path.join(opts.save_dir, 'epoch-{}.pt'.format(epoch))
            )
</code></pre>
<pre><code class="lang-auto">def get_inner_model(model):
    return model.module if isinstance(model, torch.nn.parallel.DistributedDataParallel) else model
</code></pre>
<p>The load code:</p>
<pre><code class="lang-auto">load_data = torch_load_cpu(model_filename)
model.load_state_dict({**model.state_dict(), **load_data.get('model', {})})
</code></pre>
<pre><code class="lang-auto">def torch_load_cpu(load_path):
    return torch.load(load_path, map_location=lambda storage, loc: storage)  # Load on CPU
</code></pre>
<p>Then I see this post,so I try</p>
<p><code>torch.load(load_path, map_location=torch.device('cpu'))</code></p>
<p>But  I still get this error:<br>
“RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.”</p>

ptrblck: <p>Your <code>get_inner_model</code> looks alright, but I don’t know what e.g. <code>baseline</code> is and if this might be raising the error.<br>
Could you post a minimal and executable code snippet in case you get stuck, please?</p>

winchest: <p>Thanks for your reply! But the project is a little bit big, I couldn’t get a small snippet for you. <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"> However, you said that the problem maybe is related to the baseline part. So I checked the baseline part and maybe you said is right! The code related to load model is down below.</p>
<pre><code class="lang-auto">baseline = RolloutBaseline(model, problem, opts)
class RolloutBaseline(Baseline):
    def state_dict(self):
        return {
            'model': self.model,
            'dataset': self.dataset,
            'epoch': self.epoch
        }

    def load_state_dict(self, state_dict):
        # We make it such that it works whether model was saved as data parallel or not
        load_model = copy.deepcopy(self.model)
        get_inner_model(load_model).load_state_dict(get_inner_model(state_dict['model']).state_dict())
        self._update_model(load_model, state_dict['epoch'], state_dict['dataset'])

baseline = WarmupBaseline(baseline, opts.bl_warmup_epochs, warmup_exp_beta=opts.exp_beta)
class WarmupBaseline(Baseline):
    def state_dict(self):
        # Checkpointing within warmup stage makes no sense, only save inner baseline
        return self.baseline.state_dict()
</code></pre>
<p>So when I save the baseline as baseline.state_dict(), I save model but not model.module! Therefore,when I try to load the whole document(with the saved baseline in it), it will try to load the entire model. Is this true? <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>
<p>If it is true, is there any solution to only load the model part without the baseline part? ( except add model.module to the baseline save part because the training process takes a long time <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20">)</p>

ptrblck: <aside class="quote no-group" data-username="winchest" data-post="19" data-topic="73513">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/ecae2f/48.png" class="avatar"> winchest:</div>
<blockquote>
<p>So when I save the baseline as baseline.state_dict(), I save model but not model.module! Therefore,when I try to load the whole document(with the saved baseline in it), it will try to load the entire model. Is this true? <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>This sounds plausible, yes.</p>
<p>But also it seems you are storing the entire model:</p>
<pre data-code-wrap="python"><code class="lang-python">    def state_dict(self):
        return {
            'model': self.model,
            'dataset': self.dataset,
            'epoch': self.epoch
        }
</code></pre>
<p>instead of its <code>state_dict</code>.</p>
<aside class="quote no-group" data-username="winchest" data-post="19" data-topic="73513">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/ecae2f/48.png" class="avatar"> winchest:</div>
<blockquote>
<p>If it is true, is there any solution to only load the model part without the baseline part?</p>
</blockquote>
</aside>
<p>Why wouldn’t you want to save the <code>baseline</code>’s <code>state_dict</code>, too?<br>
I guess something like:</p>
<pre data-code-wrap="python"><code class="lang-python">    def state_dict(self):
        return {
            'model': get_inner_model(self.model).state_dict(),
            'dataset': self.dataset,
            'epoch': self.epoch
        }
</code></pre>
<p>might work.</p>

winchest: <p>Thanks for your reply!! If I save the baseline’s state_dict, I have to train the model again and the training process takes a long time <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>