Are instance variables shared or re-initialized in DDP mode?

ayushexel: <p>I’m building a Trainer class that supports pytorch DDP(multi-gpu training). The structure is like this</p>
<pre><code class="lang-auto">class trainer:
    def __init__(self, ...):
        # initialize once here
        self.train_loader = ...
        self.model = ....
        self.optimizer = ...
    
    def train_loop(self, rank, world_size):
        # transfer to current rank
        self.model.to(rank)
        for data in train_loader:
            data = data.to(rank)

    def run(self):
    # spawn processes
    world_size = ...
    torch.multiprocessing.spawn(self.train_loop,
                               args=(world_size,),
                      nprocs=world_size, join=True)
</code></pre>
<p>So the above code works. But no syntax error doesn’t mean that it’s doing what it’s supposed to do, especially with DDP.</p>
<p>Question:</p>
<ul>
<li>Is the approach correct? I think that, unlike threading, multi-processing will spawn independent processes and all of them will have their copy of the trainer object. Is that correct?</li>
</ul>

aazzolini: <aside class="quote no-group" data-username="ayushexel" data-post="1" data-topic="162401">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ayushexel/48/50603_2.png" class="avatar"> ayushexel:</div>
<blockquote>
<ul>
<li>Is the approach correct? I think that, unlike threading, multi-processing will spawn independent processes and all of them will have their copy of the trainer object. Is that correct?</li>
</ul>
</blockquote>
</aside>
<p>Yes, you are right. You will get , for each rank, a completly independent instance of your class trainer, with all of it’s variables. Moreover, as a general rule, you won’t be able to share data / instances across ranks, since they live on different processes.</p>

ayushexel: <p>Thanks so much. I just wanted to confirm this. I was able to build the trainer and DDP is working correctly as I can verify the speedup. But there is one strange thing - there are e extra rank 0 processes being created. Even with 4 GPUs, the rank 0 GPU gets 2 extra processes. Not sure why. Maybe you can think of something? Thanks!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffc3446fc0ec74205e6889878f9a1351a88a65f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ffc3446fc0ec74205e6889878f9a1351a88a65f6" title="193218020-79949b4b-7baf-4d39-b643-71e537a05278"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffc3446fc0ec74205e6889878f9a1351a88a65f6_2_584x500.png" alt="193218020-79949b4b-7baf-4d39-b643-71e537a05278" data-base62-sha1="AuA71uc5Zcpyx4Z3an5fMheWuBo" width="584" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffc3446fc0ec74205e6889878f9a1351a88a65f6_2_584x500.png, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffc3446fc0ec74205e6889878f9a1351a88a65f6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffc3446fc0ec74205e6889878f9a1351a88a65f6.png 2x" data-dominant-color="272A28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">193218020-79949b4b-7baf-4d39-b643-71e537a05278</span><span class="informations">796×681 175 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>