Multi-Threaded Backprop Failing in A3C Implementation

Merlin136: <p>Greetings,</p>
<p>I am attempting to implement the A3C algorithm - requiring that there be a pool of agents, each undertaking a relatively distinct trajectory through the environment. This is accomplished by using the torch.multiprocessing module.</p>
<p>I am receiving a relatively cryptic error, requesting that I report a bug to Pytorch:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/9960a8edb3b300340c8c20d695e74d3d7dfd61ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/9960a8edb3b300340c8c20d695e74d3d7dfd61ec" title="Screenshot from 2022-07-21 12-15-52"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9960a8edb3b300340c8c20d695e74d3d7dfd61ec_2_690x384.png" alt="Screenshot from 2022-07-21 12-15-52" data-base62-sha1="lSQ92uE3MyNoJdHGTyxFCoWkZgo" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9960a8edb3b300340c8c20d695e74d3d7dfd61ec_2_690x384.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9960a8edb3b300340c8c20d695e74d3d7dfd61ec_2_1035x576.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9960a8edb3b300340c8c20d695e74d3d7dfd61ec_2_1380x768.png 2x" data-dominant-color="363635"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-07-21 12-15-52</span><span class="informations">1956×1089 308 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The actual error seems to consist somewhere in the backpropagation of the multi threaded loss function. The following is the function that is to be “multiplexed” - if you will. See the backprop error at the “HERE” comment:</p>
<pre><code class="lang-auto">def worker(name, input_shape, n_actions, global_agent, optimizer, env_id,
    n_threads, global_idx):
    T_max = 20
    local_agent = ActorCriticNetwork(input_shape, n_actions)
    memory = Memory()

    # input shape has # channels first but the frame buffer must have # channels last
    # swap the channels:
    frame_buffer = [input_shape[1], input_shape[2], 1]
    env = make_environment(env_id, shape = frame_buffer)

    episode, max_eps, t_steps, scores = 0, 1000, 0, []

    while episode &lt; max_eps:
        obs = env.reset()
        score, done, ep_steps = 0, False, 0
        hidden_state = T.zeros(1, 256)
        while not done:

            obs = T.tensor([obs], dtype = T.float)# obs = T.tensor(np.array(obs), dtype = T.float) # CURRENT

            action, value, log_prob, hidden_state = local_agent(obs, hidden_state)
            next_obs, reward, done, info = env.step(action)
            memory.store_transition(reward, value, log_prob)
            score += reward
            ep_steps += 1
            t_steps += 1
            obs = next_obs

            if ep_steps % T_max == 0 or done:
                rewards, values, log_probs = memory.sample_memory()
                loss = local_agent.calc_cost(
                    obs, hidden_state, done, rewards, values, log_probs
                )
                optimizer.zero_grad()
                hidden_state = hidden_state.detach_()
                # hidden_state = hidden_state.detach() # CURRENT


                loss.backward() ### ------------------------ HERE ------------------------ ###


                # loss.sum().backward() # CURRENT
                T.nn.utils.clip_grad_norm_(local_agent.parameters(), 40)

                for local_param, global_param in zip(
                                                local_agent.parameters(),
                                                global_agent.parameters()):
                    global_param._grad = local_param.grad

                optimizer.step()
                local_agent.load_state_dict(global_agent.state_dict())
                memory.clear_memory()

        episode += 1

        with global_idx.get_lock():
            global_idx.value += 1

        if name == '1':
            scores.append(score)
            avg_score = np.mean(scores[-100:])
            print(f'A3C episode: {episode}, thread: {name} of {n_threads}, steps: {t_steps/1e6}, score: {score}, avg_score: {avg_score}')

    if name == '1':
        x = [z for z in range(episode)]
        plot_learning_curve(x, scores, 'A3C_pong_final.png')
</code></pre>
<p>I’ve scoured around for a while and found nothing particularly helpful. I’m happy to elaborate and to provide additional material/context. Any assistance will be greatly appreciated.</p>
<p>Many thanks,</p>
<p>Merlin <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Could you check if you are seeing the same error in the latest nightly binary?<br>
If so, would it be possible for you to post a minimal, executable code snippet to reproduce the issue?</p>

Merlin136: <p>Thanks for the reply.</p>
<p>I assume I just install the latest nightly PyTorch version into a virtual environment and try again?<br>
I’ll try this anyway.</p>
<p>Thanks again.</p>

Merlin136: <p>Hmmm,</p>
<p>So I attempted this via running the latest nightly binary for the cpu version of PyTorch - in a virtual environment. Here is the exact version I downloaded:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/4/8426d80a1293eb14b9c12213b16a3fd24ce8343b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8426d80a1293eb14b9c12213b16a3fd24ce8343b" title="Screenshot from 2022-07-24 13-50-17"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/8426d80a1293eb14b9c12213b16a3fd24ce8343b.jpeg" alt="Screenshot from 2022-07-24 13-50-17" data-base62-sha1="iR4fqj1UA5pWNszk3kdAueKryor" width="690" height="264" data-dominant-color="3B1E1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-07-24 13-50-17</span><span class="informations">1075×412 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I appear to have got the exact same error as with my base install:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4fb6ba933d66f2e1ab68d9a624e92f903aabaee4.png" data-download-href="https://discuss.pytorch.org/uploads/default/4fb6ba933d66f2e1ab68d9a624e92f903aabaee4" title="Screenshot from 2022-07-24 14-47-01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fb6ba933d66f2e1ab68d9a624e92f903aabaee4_2_690x378.png" alt="Screenshot from 2022-07-24 14-47-01" data-base62-sha1="bnbdjORZrDSKknVeJnBbMuatJYM" width="690" height="378" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fb6ba933d66f2e1ab68d9a624e92f903aabaee4_2_690x378.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fb6ba933d66f2e1ab68d9a624e92f903aabaee4_2_1035x567.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fb6ba933d66f2e1ab68d9a624e92f903aabaee4_2_1380x756.png 2x" data-dominant-color="363635"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-07-24 14-47-01</span><span class="informations">1920×1053 294 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m not sure how to post a “minimal” code snippet for reproducibility, there are several dependencies for this project. Perhaps I could make a github repo for you to examine?</p>
<p>Many thanks!</p>
<p>Merlin</p>

Merlin136: <p>Actually, it seems I have managed to resolve the issue.</p>
<p>I don’t know why I didn’t try this sooner - since the error said: “No grad_fn for non-leaf saved tensor”. I just went in to the function that computes the loss and added “requires_grad=True” to the actual loss tensor, prior to computing loss.backward(). This seems to have fixed my issue, see below for the fix - at the “HERE” comment:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77151ef604a204a73f24c4af114f2e73f770736a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/77151ef604a204a73f24c4af114f2e73f770736a" title="Screenshot from 2022-07-24 17-07-44"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77151ef604a204a73f24c4af114f2e73f770736a_2_532x500.jpeg" alt="Screenshot from 2022-07-24 17-07-44" data-base62-sha1="gZs4mAksXDBkPGZYdofaCbvKy14" width="532" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77151ef604a204a73f24c4af114f2e73f770736a_2_532x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77151ef604a204a73f24c4af114f2e73f770736a_2_798x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/7/77151ef604a204a73f24c4af114f2e73f770736a.jpeg 2x" data-dominant-color="070808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-07-24 17-07-44</span><span class="informations">956×898 72.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Many thanks to anyone who took the time to read my original query.<br>
I welcome any suggestions for improvements.</p>
<p>Thanks,</p>
<p>Merlin</p>

ptrblck: <p>Your fix could avoid running into the error, but is detaching the computation graph so your model would most likely not learn anything assuming <code>total_loss</code> is used to compute the gradients.</p>

Merlin136: <p>Yeah this is exactly what happened.</p>
<p>total_loss is used in precisely that way, and the model learned nothing.<br>
Why would specifying that the gradient is required, in any way detach it? Is there a simple work-around from here?</p>
<p>Thanks again.</p>

vmoens: <p>I think your log_probs are sampled from the replay buffer aren’t they? They should be differentiable, which is almost surely not the case if they come from the RB. Have you considered re-computing them by passing through the policy network when calculating the loss? I think this may be the reason why the backward does not produce gradients for the policy weights. As for the critic weights they should be ok since you call self.forward (unless done is True, I would make sure that done is mostly False in your task).</p>
<p>If you’re interested I think <a href="https://github.com/facebookresearch/rl" rel="noopener nofollow ugc">torchrl</a> provides most of the primitives you’re looking for here, we’d be glad to provide support if you’d like to use them.</p>

ptrblck: <aside class="quote no-group" data-username="Merlin136" data-post="7" data-topic="157132">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/merlin136/48/51387_2.png" class="avatar"> Merlin136:</div>
<blockquote>
<p>Why would specifying that the gradient is required, in any way detach it? Is there a simple work-around from here?</p>
</blockquote>
</aside>
<p>You are recreating a new tensor which would detach it from the current computation graph and creates a new leaf-tensor. The right approach would be to use the output tensors directly without rewrapping them.</p>
<p><a class="mention" href="/u/vmoens">@vmoens</a>’s suggestion sounds valid and <code>torchrl</code> might be a good starter for your use case.</p>

Merlin136: <p>Thanks vmoens,</p>
<p>Yes, my log_probs are sampled from a replay buffer; they consist in a python list of pytorch tensors. When it comes time to compute the loss I cast this list to a pytorch tensor with torch.cat:</p>
<pre><code class="lang-auto">log_probs = T.cat(log_probs)
</code></pre>
<p>After inspecting the gradient function assigned to this subsequent tensor, I get this:</p>
<pre><code class="lang-auto">log_probs.grad_fn: &lt;CatBackward0 object at 0x7f1a49496880&gt;
</code></pre>
<p>So at leas there is a gradient function for Backward to use when computing the loss. Assuming that this function is differentiable, I can’t see what the problem might be with backward.</p>
<p>I soon realised that there was no need to cast total_loss to another tensor in which requires_grad is true; since this is the default behaviour anyway (whoops). Thus - I think - I won’t run into the detaching issue that <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned.</p>
<p>Here is what my loss function looks like now:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/127116e3058bf1de479eee4e712eaac992e3688e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/127116e3058bf1de479eee4e712eaac992e3688e" title="Loss_fn"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/127116e3058bf1de479eee4e712eaac992e3688e_2_656x500.jpeg" alt="Loss_fn" data-base62-sha1="2D8T18IKSezyzKJRdT3VbAJ8Oiq" width="656" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/127116e3058bf1de479eee4e712eaac992e3688e_2_656x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/127116e3058bf1de479eee4e712eaac992e3688e_2_984x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/2/127116e3058bf1de479eee4e712eaac992e3688e.jpeg 2x" data-dominant-color="070808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Loss_fn</span><span class="informations">1176×896 178 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for the suggestion to check out torchrl. I have not done so as of yet (this project has had to hit the back burner for the last week) but I endeavour to have a look.</p>
<p>I’m not sure what you mean exactly, regarding your suggestion to recalculate the log_probs when it comes time to use them in the loss calculation. I can’t yet see how this doesn’t defeat the purpose of a replay buffer. can you elaborate on this?</p>
<p>Many thanks to anyone reading this.</p>

vmoens: <p>Usually log_probs is not sampled from the replay buffer unless it’s used to correct for the change in model configuration (see PPO and related)<br>
I suspect that if you sample data from the replay buffer you are in an off policy setting. As such, the data is not generated according to the current policy and it wouldn’t make sense to backpropagate through a graph that is now obsolete.<br>
Hence you should sample what is needed to compute the log prob from the replay buffer and backpropagate through a loss computed using a log_prob computed from the data sampled from the replay buffer.</p>
<p>Now of course this is all based on common assumptions about your model and replay buffers. If you’re storing data that is part of a graph in the replay buffer, you’re probably not using it right (usually replay buffers are like datasets: it’s fixed, graph-free data that is purely used to train a model in an off policy manner).</p>
<p>Let me know if I got anything wrong. Still happy to help if I do!</p>