What is reinitialization and why is that bad?

joshkim: <p>Hi,</p>
<p>I’m currently writing a distributed training script where there is one Trainer instance that can call its train function multiple times. I wrote this so that I can train a model with different settings sequentially (using a different optimizer, for instance).</p>
<p>The code looks something like this:</p>
<pre><code class="lang-auto">class Trainer:
    def __init__(self, loss_fn, device, dataset):
        self.device = device
        self.loss_fn = loss_fn
        self.dataset = dataset
...


    def train(self, epochs):
        torch.cuda.set_device(self.device)
        init_process_group(backend="nccl")

       ...training logic here...

        barrier()  # minimize the chance of the next train function starting before other processes finish
        destroy_process_group()
        sleep(10)

if __name__ == "__main__":
    dataset = CustomDataset(10000)

    loss_fn = nn.CrossEntropyLoss()

    trainer = Trainer(
        loss_fn=loss_fn,
        device=int(os.environ.get("LOCAL_RANK", 0)),
        dataset=dataset,
    )
    trainer.train(epochs=5)
    trainer.train(epochs=5)
    trainer.train(epochs=5)
    trainer.train(epochs=5)
    trainer.train(epochs=5)
    trainer.train(epochs=5)
</code></pre>
<p>The problem is I often run into the following error on my 2nd or 3rd execution of <code>trainer.train()</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/3445394437987088710b70336cde533afa2116a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/3445394437987088710b70336cde533afa2116a6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3445394437987088710b70336cde533afa2116a6_2_690x82.png" alt="image" data-base62-sha1="7sp8EqNj7AlDZw9cjpwEQFaTzdI" width="690" height="82" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3445394437987088710b70336cde533afa2116a6_2_690x82.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3445394437987088710b70336cde533afa2116a6_2_1035x123.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3445394437987088710b70336cde533afa2116a6_2_1380x164.png 2x" data-dominant-color="303131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1472×176 26.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m aware that “reinitialization is not recommended” based on the following doc:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96be1a71c20b569a41a4f8c6bf103bfaeb0f7d10.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/96be1a71c20b569a41a4f8c6bf103bfaeb0f7d10" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96be1a71c20b569a41a4f8c6bf103bfaeb0f7d10_2_690x165.jpeg" alt="image" data-base62-sha1="lvwUJhrE5Agk5orEw29rAA5oDwA" width="690" height="165" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96be1a71c20b569a41a4f8c6bf103bfaeb0f7d10_2_690x165.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/9/6/96be1a71c20b569a41a4f8c6bf103bfaeb0f7d10.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/6/96be1a71c20b569a41a4f8c6bf103bfaeb0f7d10.jpeg 2x" data-dominant-color="EEEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">800×192 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, I’ve used some safe guards like barrier and sleep to increase the chance of destroying the previous default process before initializing a new default process group. The NCCL debug logs also indicate the process group is destroyed as well before that DDP error above.</p>
<p><strong>Question 1:</strong><br>
The DDP error still persists no matter how long I put it to sleep. Why is that the case?</p>
<p><strong>Question 2:</strong><br>
There must be a way to properly reinitialize since things like <code>torchrun</code> probably has to do reinitialization for fault-tolerant training. How does it make sure the reinitialized code doesn’t produce the error like mine??</p>