Process group doesn't terminate by itself?

username: Wenhao-Yang
cooked: <p>Has anyone encountered a similar problem?</p>
<p><strong>The Python script could not exit by itself without Ctrl+C.</strong> I try to train a model on 2 GPU in a ubuntu server.</p>
<p>Training and evaluation are fine. But after “main()” finishing, the python process won’t terminate by itself. I checked the gpu( <code>nvidia-smi</code> ) and cpu usage ( <code>htop</code> ). The process still occupied those resources, which means the process group could not be kill by itself. And if I training with only one GPU, it’s fine.</p>
<p>Here’s my opinions:</p>
<ul>
<li>
<code>group_process</code> could not terminated for some reason? Except for initializaiton by <code>torch.distributed.init_process_group()</code>, I didn’t find any solutions in torch Documentation to termanate the group.</li>
<li>Deleting variable like <strong>model</strong> could not help it out.</li>
<li>Is this a bug for torch or python ?</li>
<li>Did I make mistakes in using multi-gpu?</li>
</ul>

username: Yanli_Zhao
cooked: <p>code snippets?</p>
<p>also, why not use init_process_group()?</p>

username: BramVanroy
cooked: <p>From what I remember you have to destroy the distributed groups, so at the end of your code:</p>
<pre><code class="lang-auto">import torch.distributed as dist
dist.destroy_process_group()
</code></pre>

username: Wenhao-Yang
cooked: <p><strong>Thanks for your advice! <img src="https://discuss.pytorch.org/images/emoji/apple/grimacing.png?v=9" title=":grimacing:" class="emoji" alt=":grimacing:"></strong><br>
I tried <code>destroy_process_group()</code> but it didn’t work.</p>
<p>In the end, I fixed it by deleting the following Potential conflict line:<br>
<code>torch.multiprocessing.set_sharing_strategy('file_system')</code></p>
<p>It seems that this line is compatible with</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend="nccl", init_method='file:///home/xxx//sharedfile', rank=0, world_size=1)
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend="nccl", init_method='tcp://localhost:32546', rank=0, world_size=1)
</code></pre>
<p>and it may cause the problems.</p>

username: Wenhao-Yang
cooked: <p>Sorry for missing code snippets. This is the first time to create a topic in the community. Here are my codes. And I have already find the potential conflict line:</p>
<pre><code class="lang-auto"># import ...
os.environ['CUDA_VISIBLE_DEVICES'] = “0,1”


def main():
    train_loader = torch.utils.data.DataLoader(train_dir,
                                                                       batch_size=args.batch_size, 
                                                                       shuffle=args.shuffle,
                                                                        **kwargs)
    
    # --------------------------------------------------- confilct line -------------------------------------------
    torch.distributed.init_process_group(backend="nccl",
                                                              init_method='tcp://localhost:32546',
                                                              rank=0, 
                                                              world_size=1)

    # -----------------------------------    line without the problem   ---------------------------------------
    torch.distributed.init_process_group(backend="nccl", init_method='file:///home/xxx/sharedfile',
                                                              rank=0,
                                                              world_size=1)

    model = DistributedDataParallel(model.cuda())

    # train()...
    # eval()...

if __name__ == '__main__':
    main()

</code></pre>
<p>Thanks, anyway! <img src="https://discuss.pytorch.org/images/emoji/apple/grimacing.png?v=9" title=":grimacing:" class="emoji" alt=":grimacing:"></p>