Deep Deterministic Policy Gradient implementation

lubiluk: <p>Hi, I want to use DDPG in my project so I set out to first get a working example. I’ve found this nice implementation in Keras (<a href="https://keras.io/examples/rl/ddpg_pendulum/" rel="nofollow noopener">https://keras.io/examples/rl/ddpg_pendulum/</a>) which works well and decided to translate it to PyTorch 1:1. Unfortunately the model doesn’t learn. Since I have no friends to ask, could someone please have a look on my code and try to spot where I’ve made a mistake?</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/lubiluk/ddpg" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/385792?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/lubiluk/ddpg" target="_blank" rel="nofollow noopener">lubiluk/ddpg</a></h3>

<p>Contribute to lubiluk/ddpg development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Do you have an idea which part of the code might be wrong or where you were unsure, how to implement a specific use case in PyTorch?</p>
<p>If not, you could try to check the model implementation first e.g. by using a constant input (tensor of all ones should work). Of course you would need to load the parameters from one model to the other and I don’t know, how hard this would be.<br>
Once the models return the same output, you could debug the code step by step.</p>

iffiX: <p>Find 1 Problem: where is terminal?:</p>
<pre><code class="lang-auto">reward + discount * ~terminal * next_value
</code></pre>
<p>Other parts seems to be ok.</p>

lubiluk: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I was most unsure about how to define a network that has parallel input processing with result concatenation.  See Critic class and it’s forward method.</p>
<p><a class="mention" href="/u/iffix">@iffiX</a> it’s an interesting point. I think you are referring to the definition of ‘y’. But in the original Keras code there is no terminal as well in that line.</p>

ptrblck: <p>The <code>Critic</code> implementation looks correct to me.<br>
Note that I’m not deeply familiar with Keras and assume that the <code>Concatenate</code> layer concatenates the input tensors in the “feature dimension”.</p>

lubiluk: <p>I was pointed out that I should model.eval() and model.train(), every time I want to call my models. It’s because I use batch normalization and it operates differently depending on train flag. Now the model learns correctly.<br>
Is it a good idea to always wrap model calls with eval/train?</p>
<pre><code class="lang-auto">model.eval()
r = model(input)
model.train()
</code></pre>

iffiX: <p>Oh, that point was neglected, “eval” and “train” are not needed if you are not using something special like<br>
<code>BatchNorm</code><br>
See <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=eval#torch.nn.Module.eval" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=eval#torch.nn.Module.eval</a></p>

ptrblck: <aside class="quote no-group" data-username="lubiluk" data-post="6" data-topic="91702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lubiluk/48/27506_2.png" class="avatar"> lubiluk:</div>
<blockquote>
<p>Is it a good idea to always wrap model calls with eval/train?</p>
</blockquote>
</aside>
<p>Yes, I would recommend to always call <code>model.train()</code> before the training and <code>model.eval()</code> before the evaluation or testing of the model. Even if your current model does not use any batchnorm or dropout layers, the model itself or custom layers might use the <code>self.training</code> attribute to change their behavior.</p>

lubiluk: <p>Somehow this doesn’t seem right:</p>
<pre><code class="lang-auto">        critic_optimizer.zero_grad()

        target_actions = target_actor(next_state_batch)
        y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
        critic_model.eval()
        critic_value = critic_model([state_batch, action_batch])
        critic_model.train()
        critic_loss = torch.mean(torch.square(y - critic_value))

        critic_loss.backward()
        critic_optimizer.step()
</code></pre>
<p>Didn’t I just exclude batch norm parameters from training effectively getting rid of normalization? I mean this is part of the training routine so it seems logical to have the model in train() mode. I have read the original Batch Norm paper and it seems to confirm that.<br>
Now I’m really confused.<br>
Just for testing I entirely commented out batch normalization and the model is training well without it.</p>
<p>So the conclusion is, there is something wrong with my batch normalization and I still don’t know what…</p>

ptrblck: <p>I missed that the previous post was related to the model implementation and not a general question.<br>
By calling <code>critic_model.eval()</code>, you would use the running stats instead of the batch stats to normalize the data. The affine parameters of the batchnorm layers would still be trained.<br>
I’m not familiar with the model, but why did you add this line of code and what is the Keras model doing?</p>

lubiluk: <p>I’ve added this line because I tried all combinations of eval and train, and learning works only when i put eval everywhere. But that probably means that batch norm is not used at all.<br>
I also tried to remove batchnorm layers altogether and it also enables learning.<br>
Keras model probably also has a slight bug as it always keeps batchnorm layer in evaluation mode. But surprisingly, when I put it in training mode then learning abilities are not affected.<br>
I am thoroughly confused right now and I will probably go carefully through the implementations to debug what’s going on. Just like you suggested in your first post. I think I need to debug my autograd chain. Is there a way to see it as a graph?</p>

iffiX: <p>Try use this:</p>
<pre><code class="lang-auto">from torchviz import make_dot

def visualize_graph(final_tensor, visualize_dir="", exit_after_vis=True):
    """
    Visualize a pytorch flow graph

    Args:
        final_tensor: The last output tensor of the flow graph
        visualize_dir: Directory to place the visualized files
        exit_after_vis: Whether to exit the whole program
            after visualization.
    """
    g = make_dot(final_tensor)
    g.render(directory=visualize_dir, view=False, quiet=True)
    if exit_after_vis:
        exit(0)
</code></pre>

lubiluk: <p>Thank you for your help. I’ve spend some more time with my models and what I’ve found out is that nn.BatchNorm1d (PyTorch) works differently than layers.BatchNormalization (Keras). There are some discussions about that on the internet but not conclusive. I can’t work out how to make them behave the same.</p>
<p>In the end I’ve replaced nn.BatchNorm1d with nn.LayerNorm and the agent learns flawlessly.</p>
<p>I would really like to know what’s wrong with nn.BatchNorm1d in PyTorch. Until I find out I’ve got to avoid it.</p>

alessio_brini: <p>Any update on this? I started using Pytorch for my RL model and I experienced too that the behavior of BatchNormalization in Tensorflow is different from nn.BatchNorm1d of PyTorch. Have you just applied LayerNorm paying attention to call .train() and .eval() in the right places?</p>
<p>I have found another <a href="https://discuss.pytorch.org/t/minimal-working-example-for-strange-behavior-of-batchnorm/22150">thread</a> where it is said that BatchNorm1d when on model.train() does not use running stats to normalize the input of the layer, which seems confusing to me. Is that mean that the layer just compute running stats and normalize input only when evaluating the model? I would say that in this case the learning would be really wrong</p>

lubiluk: <p>Actually, I think it turned out that the problem was in Keras… I have reported an issue there and they have removed batch normalization from the example in question, because it prevented the model from learning.<br>
Please see this thread: <a href="https://github.com/keras-team/keras-io/issues/198" class="inline-onebox" rel="noopener nofollow ugc">DDPG example uses BatchNormalization incorrectly · Issue #198 · keras-team/keras-io · GitHub</a>.</p>
<p>And about your question, batch norm normalizes differently during training and inference. During training stats of current batch are used, during inference, moving stats of all seen batches are used.</p>
<p>Training:<br>
<code>(batch - mean(batch)) / (var(batch) + epsilon) * gamma + beta</code><br>
Inference:<br>
<code>(batch - self.moving_mean) / (self.moving_var + epsilon) * gamma + beta</code><br>
Source: <a href="https://keras.io/api/layers/normalization_layers/batch_normalization/" class="inline-onebox" rel="noopener nofollow ugc">BatchNormalization layer</a></p>
<p>In my view, PyTorch docs say the same, although maybe in a bit less concise form<br>
<code>..Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation...</code><br>
Source: <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm1d — PyTorch 1.8.1 documentation</a></p>
<p>In my project I also removed batch norm, and layer norm too. Although layer norm worked well, but pure MLP without any normalization worked too so why complicate. I’ve read somewhere that Layer norm in general behaves better than BatchNorm</p>