DQN example from PyTorch diverged!

Eddie_Li: <p>Hello folks.</p>
<p>I just implemented my DQN by following the example from PyTorch. I found nothing weird about it, but it diverged. I run the original code again and it also diverged.</p>
<p>The behaviors are like this. It often reaches a high average (around 200, 300) within 100 episodes. Then it starts to perform worse and worse, and stops around an average around 20, just like some random behaviors. I tried a lot of changes, the original version was surprisingly the best one, as described.</p>
<p>Any ideas?</p>

Eddie_Li: <p>Another concern is more theoretical. Since I’m training it on CartPole, in which sequential or temporal information matters, would DQN be able to learn the best strategy, if enough training given?</p>

stegben: <p>Hi, I just submit a PR of DQN in PyTorch:<br>
<aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/examples/pull/172" target="_blank" rel="nofollow noopener">github.com/pytorch/examples</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/examples/pull/172" target="_blank" rel="nofollow noopener">DQN</a>
    </h4>

    <div class="branches">
      <code>pytorch:master</code> ← <code>stegben:dqn</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-06-17" data-time="16:40:13" data-timezone="UTC">04:40PM - 17 Jun 17 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/stegben" target="_blank" rel="nofollow noopener">
          <img alt="stegben" src="https://avatars3.githubusercontent.com/u/6868283?v=4" class="onebox-avatar-inline" width="20" height="20">
          stegben
        </a>
      </div>

      <div class="lines" title="2 commits changed 2 files with 180 additions and 0 deletions">
        <a href="https://github.com/pytorch/examples/pull/172/files" target="_blank" rel="nofollow noopener">
          <span class="added">+180</span>
          <span class="removed">-0</span>
        </a>
      </div>
    </div>

  </div>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I’ve encounter that problem before. Maybe you use the same Q network when updating? The Q network for finding the greatest action should be fixed.</p>

Eddie_Li: <p>Thanks man.</p>
<p>I tried after I saw your reply. The nature DQN paper didn’t mention how frequently they updated their target network. But I tried every 2, 10, and 100 iterations. None of them solved my divergence problem. But they made my averages stay at 200 for a very long time. I guess it’s working, but not enough.</p>
<p>BTW I’m using exactly the same implementation given by PyTorch tutorials, which takes screens as inputs.</p>

xuehy: <p>How do you keep the average over 200 for a period? In my experiments, it cannot keep. There is only bursts into high durations and the following duration will fall to very low values. I also tried the the update frequency but it does not work. Can you show me your duration-episode curve?</p>
<p>I am also using the difference image as the state.</p>

Eddie_Li: <p>Hey xuehy. Sorry since it was not very successful so I didn’t keep a lot of records. What I was doing was just adjusting hyperparameters. What I do remember is, memory capacity matters a lot.</p>
<p>So far I haven’t found a solution to make DQN converge. The potential reasons in my mind are 1) NN models are prone to divergence in theory; 2) simple image recognition model doesn’t learn CartPole very well.</p>
<p>I don’t know how far did you get since then, but I would suggest both include a LSTM cell, and try your model on other games.  I haven’t done them yet though.</p>

j.laute: <p>I found that the usage of smooth l1 loss (Huber) always led to divergence on the cart pole environment (somebody else also had that problem I’ll add the link later)</p>
<p>It worked for me when using l2 (mse) loss</p>
<p>Further investigation showed that the loss explodedn with the Huber loss, (I ll need to check what happened to the gradient)</p>
<p>I found this very strange considering that the Huber loss is specifically designed to be stable</p>
<p>If anyone has any idea why this happens please let me know</p>
<p>Tldr if you usw smoothl1 loss, just try mse</p>

Eddie_Li: <p>Thanks for pointing out! But I still can’t make it converge. In my case it just diverges very slowly (in case that my hyperparameters are problematic, I use the original example).</p>
<p>Can you let us know what hyperparameters and other tricks, like fixed Q target, you are using?</p>

tessler: <p>Hey,<br>
You can take a look at the original implementation (beware, it’s in LUA): <a href="https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner" rel="nofollow noopener">https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner</a><br>
There are several tricks they use:</p>
<ol>
<li>They play using the static network, the dynamic one it trained, every 10K steps (hyper-parameter called target_q) the static is replaced by the dynamic.</li>
<li>Reward clipping - reward is clipped to +- 1</li>
<li>They clip the delta (loss) to +- 1 also.</li>
<li>Probably some more tricks I forgot / missed.</li>
</ol>
<p>Besides that, once the basic DQN is working, I would suggest to add in Double Q Learning. It is pretty straight-forward to implement and gives steady convergence and better results.</p>

j.laute: <p><a class="mention" href="/u/eddie_li">@Eddie_Li</a> I’ll look for the code and put it on github (it’s a mess though). I was not able to find the blog post stating that the Huber loss leads to divergence on the cart pole example <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>

j.laute: <p>Here is the github link <a href="https://github.com/jaidmin/pytorch-q-learning" rel="nofollow noopener">https://github.com/jaidmin/pytorch-q-learning</a></p>
<p>I just pushed my q-learning folder to github, the cartpole example is in “pytorch-q-learning-simple.ipynb”, that<br>
it converges for me after ~ a couple hundred / maybe 1-2 thousend episodes.<br>
As you can see I use fixed q-targets and a replay memory</p>
<p>I’ll create a gist with a simple minimal example later</p>
<p>If you find any errors in the code or have any questions please let me know</p>
<p>johannes</p>

xuehy: <p>I just changed the state from the image difference to the original state of the game environment. After tuning the learning rate, everything is fine now. It seems the image input is difficult to train.</p>

denizs: <p>Definitely, it’s way harder to train than training your model with the observations, which in that particular case of <code>CartPole-v0</code> return the underlying physical model of the game. However, it is also way cooler, as it demonstrates the power of this algorithm <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>However, I’d like to mention that the original Deep Q Learning algorithm described by <a href="https://www.google.de/search?q=deep+q+learning+atari&amp;oq=deep+q+learning+atari&amp;aqs=chrome..69i57.4568j0j7&amp;sourceid=chrome&amp;ie=UTF-8" rel="nofollow noopener">Mnih et al.</a> didn’t use the difference of previous and current frame as a state representation, but rather a stack of the 4 last seen and processed frames (Resulting in a 1x4x84x84 input tensor as they were training on gray scale images). Also, they leveraged a technique called frame skipping:</p>
<p>The agent selects an action on every kth frame (I think it was 4th in the paper) and  this action is applied to the environment for k-1 frames. This helps the agent to see more frames in the same time, as computing an action requires significantly more computing power than stepping the environment.</p>
<p>Also, the paper mentioned priorly deployed a second ‘target network’, which would be updated every 10.000 frames, which additionally stabilizes the algorithm.</p>

vv_ss: <p>xuehy, I am facing the same problem on CartPole… the agent learns the most optimal behavior after 2K episodes, but then wanders off into less optimal regions. Can you please provide your learning rate, discount factor, exploration epsilon that worked for you? Thanks!</p>

xuehy: <p>Well. Learning doesn’t take such long time. I only train it until the 300th episode. The result is stable after around 100th episode. The discount factor is 0.9999. The learning rate is 0.01 with adam. the exploration epsilon starts at 0.9 and ends at 0.05 and the eps_decay is 200.</p>

Eddie_Li: <p>I’m afraid people like me who trained on pixels can’t make it work because the example is wrong on its preprocessing. See denizs’ post.</p>

mufeili: <p>Double DQN might help. DQN is known to have a very high variance and there is no guarantee of convergence. Double DQN yields a much lower variance and a better policy from my experiments on CartPole. While I used state signals from Gym directly, this might also work for the images.</p>

AlbertM: <p>Hi! I think the preprocessing makes sense, but not very well, as the difference of the two frames capture the following information:</p>
<ol>
<li>the current position and the next position</li>
<li>which position is current, and which position is next</li>
</ol>
<p>However, it does not capture the absolute position very well, because it uses a region of interest, which will keep fixed when the cartpole is in the middle. I think this might be where divergence comes.</p>

mimoralea: <p>I wanted to add my two cents.</p>
<p>One of the problems is related to what is known as the “vanishing gradient” problem in supervised learning. Bottom line is, your agent learns a good policy and then stops exploring the areas that are sub-optimal. At one point, all your agent knows about is the best states, and actions because there is nothing other than the samples coming from a near-optimal policy in your replay buffer. Then, all updates to your network come from the same couple of near-optimal states, actions.</p>
<p>So, due to the vanishing gradient problem, your agent forgets how to get to the best, straight-up pole, position. It knows how to stay there, but not how to get there. You have no more samples of that in your replay buffer, guess what. As soon as the initial state is minimally different, chaos…</p>
<p>BTW, this happens in virtually any DQN/Cart Pole example I’ve tested, even the ones using the continuous variables as opposed to images. Yes, this includes OpenAI Baselines! Just change the code so it keeps training indefinitely and you’ll find the same divergence issues.</p>
<p>The way I got it to perform better is to increase the Replay Buffer size to 100,000 or 1,000,000 (you may need a solid implementation - see OpenAI Baselines: <a href="https://github.com/openai/baselines/blob/master/baselines/deepq/replay_buffer.py" rel="noopener nofollow ugc">https://github.com/openai/baselines/blob/master/baselines/deepq/replay_buffer.py</a>), and increase the Batch Size to ~64 or ~128. I think reducing the learning rate would help as well, but also slow down learning, of course. Though, I suppose this will only postpone the issues, but at least I ran 1,000 episodes of perfect performance which works for me.</p>
<p>Finally, I found it interesting to review the basics described by Sutton in his book: <a href="http://incompleteideas.net/book/the-book-2nd.html" class="inline-onebox" rel="noopener nofollow ugc">Sutton &amp; Barto Book: Reinforcement Learning: An Introduction</a></p>
<p>From the book, take a look at Example 3.4: Pole-Balancing.</p>
<blockquote>
<p>Example 3.4: Pole-Balancing The objective in this task is to apply forces to a cart moving along<br>
a track so as to keep a pole hinged to the cart from falling over: A failure is said to occur if the pole<br>
falls past a given angle from vertical or if the cart runs off the track. The pole is reset to vertical<br>
after each failure. This task could be treated as episodic, where the natural episodes are the repeated<br>
attempts to balance the pole. The reward in this case could be +1 for every time step on which failure<br>
did not occur, so that the return at each time would be the number of steps until failure. In this case,<br>
successful balancing forever would mean a return of infinity. Alternatively, we could treat pole-balancing<br>
as a continuing task, using discounting. In this case the reward would be −1 on each failure and zero<br>
at all other times. The return at each time would then be related to −γ<br>
K, where K is the number of<br>
time steps before failure. In either case, the return is maximized by keeping the pole balanced for as<br>
long as possible.</p>
</blockquote>
<p>As the Cart-Pole example is setup as an episodic task in OpenAI Gym; +1 every time step on which failure did not occur, gamma should then be set to 1.</p>

alexis-jacq: <p>And if you want to explore what it the list of tricks that make the cartpole balancing faster and longer with Pytorch, from simple DQN to Double Duelling DQN, I recommend this tutorial:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/criteo-research/paiss_deeprl/blob/master/exercises_pytorch.ipynb" target="_blank">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/criteo-research/paiss_deeprl/blob/master/exercises_pytorch.ipynb" target="_blank">criteo-research/paiss_deeprl/blob/master/exercises_pytorch.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PAISS Practical Deep-RL by Criteo Research (Pytorch version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "\n",
    "from utils import RLEnvironment, RLDebugger\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/criteo-research/paiss_deeprl/blob/master/exercises_pytorch.ipynb" target="_blank">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>