Data Distributed Parallel Out of Sync

Matthew_Landen: <p>Recently, I have been running training of deep reinforcement learning policy network in a distributed manner with ddp. I am observing confusing behavior where the different ranks are at different learning updates at the same time. This does not m ake sense to me because I would think that the first call to loss.backward would wait for all ranks to get through the first batch before returning and continuing on to optimizer step.</p>
<p>A code snippet is below.</p>
<pre><code class="lang-python">with contextlib.ExitStack() as stack:
            if self.problem.agent_train:
                stack.enter_context(self.agent_model.join())
            if self.problem.detector_train:
                stack.enter_context(self.detection_model.join())
            if self.problem.attacker_train:
                stack.enter_context(self.attacker_model.join())

 # Main training loop
        train_iterations = [0 for _ in memories]
        last_checkpoints = [time.time() for _ in memories]

        error = torch.zeros(1)
        error_signal = None
        if self.world_size  &gt; 1:
            error_signal = dist.irecv(error, tag = 1)
        else:
            error_signal = None

        self.learning_steps_completed = 0
        while self.learning_steps_completed &lt; self.n_learning_steps:
            # Check for timeout
            if (time.time() - start) &gt; self.timeout - 120:
                print("User defined timeout reached... terminating training")
                break

            # Stop learning when an error occurs in an agent
            if self.error_event.is_set():
                had_error = True
                if self.verbose &gt; 0:
                    print(f"[Rank {self.rank}] Error in agent", flush = True)
                for i in range(self.world_size):
                    if i != self.rank:
                        dist.send(error, dst = i, tag = 1)
                break

            if error_signal is not None and error_signal.is_completed():
                had_error = True
                print(f"[Rank {self.rank}] Ending early from error", flush = True)
                break

            # memories = [agent_memory, detection_memory, attacker_memory]
            # so this loop is just over the controller, detector, and attacker

            for i, memory in enumerate(memories):
                train_type, memory = memory
                if memory.has_batch():
                        self.learning_steps_completed += 1

                    # Sample the replay memory to get a batch of experiences (defined in child class of Memory)
                    batch, indexes, weights = memory.sample()

                    # Compute algorithm-specific loss (defined in child class)
                    loss, priorities = self.compute_loss(batch, weights, train_type)

                    # Do optimization step and increment the number of training iterations
                    self.optimizers[train_type].zero_grad()
                    loss.backward()
                    # dist.barrier() 
                    self.optimizers[train_type].step()
                    train_iterations[i] += 1

                    # Some periodic collective operations to collect intermediate results 
</code></pre>
<p>In the above code, the loss.backwards() is hit at different learning_steps_completed without the barrier but not with.</p>

pritamdamania87: <aside class="quote no-group" data-username="Matthew_Landen" data-post="1" data-topic="131471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matthew_landen/48/30700_2.png" class="avatar"> Matthew_Landen:</div>
<blockquote>
<p>In the above code, the loss.backwards() is hit at different learning_steps_completed without the barrier but not with.</p>
</blockquote>
</aside>
<p>Could you explain what you mean by this? Are you printing <code>self.learning_steps_completed</code> after <code>loss.backward</code> on each process and you see different values across trainers where some are running ahead of others?</p>
<p>Note that if you are using GPUs here, gpu execution is async so essentially your training loops is just enqueuing work to the GPU and when <code>loss.backward</code> returns it means the work has been enqueued to the GPU but not necessarily finished. When you do <code>dist.barrier()</code> it forces a CUDA synchronization and blocks the host until all GPU work is done.</p>

Matthew_Landen: <aside class="quote no-group" data-username="pritamdamania87" data-post="2" data-topic="131471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<p>Could you explain what you mean by this? Are you printing <code>self.learning_steps_completed</code> after <code>loss.backward</code> on each process and you see different values across trainers where some are running ahead of others?</p>
</blockquote>
</aside>
<p>Yes, I’m seeing different learning steps on different ranks at once. A max difference is about 40.<br>
I am not using a GPU at all, just cpu. Does the not ddp distributed calls cause problems for the context manager of join()?</p>

pritamdamania87: <aside class="quote no-group" data-username="Matthew_Landen" data-post="3" data-topic="131471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matthew_landen/48/30700_2.png" class="avatar"> Matthew_Landen:</div>
<blockquote>
<p>Yes, I’m seeing different learning steps on different ranks at once. A max difference is about 40.<br>
I am not using a GPU at all, just cpu.</p>
</blockquote>
</aside>
<p>Could it be possible this is just an output buffering issue? Could you add a <code>sys.stdout.flush()</code> line after your print statements and see if the output makes sense?</p>
<blockquote>
<p>Does the not ddp distributed calls cause problems for the context manager of join()?</p>
</blockquote>
<p>Not sure I followed this, are you referring to lines like <code>self.agent_model.join()</code> in your code? If so, is this the <code>ddp.join()</code> method or is this <code>join()</code> method doing something else?</p>

Matthew_Landen: <aside class="quote no-group" data-username="pritamdamania87" data-post="4" data-topic="131471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<p>Could it be possible this is just an output buffering issue? Could you add a <code>sys.stdout.flush()</code> line after your print statements and see if the output makes sense?</p>
</blockquote>
</aside>
<p>No, it couldn’t be. I already flush the output.</p>
<aside class="quote no-group" data-username="pritamdamania87" data-post="4" data-topic="131471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<p>Not sure I followed this, are you referring to lines like <code>self.agent_model.join()</code> in your code? If so, is this the <code>ddp.join()</code> method or is this <code>join()</code> method doing something else?</p>
</blockquote>
</aside>
<p>Yes, it is the ddp.join(). Because there could be three model training together, I have to enter the the model join contexts.</p>

pritamdamania87: <aside class="quote no-group" data-username="Matthew_Landen" data-post="5" data-topic="131471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matthew_landen/48/30700_2.png" class="avatar"> Matthew_Landen:</div>
<blockquote>
<p>Yes, it is the ddp.join(). Because there could be three model training together, I have to enter the the model join contexts.</p>
</blockquote>
</aside>
<p>If you are using <code>ddp.join()</code>, I’m assuming this is because you have uneven data across your trainers. If so, isn’t it expected to see different number of training steps across different trainers?</p>
<p>Could you also provide a complete repro script that we can try on our end and also the output that you are seeing on your end. Would be much easier to troubleshoot this issue for us that way.</p>