My DQN agent is not learning

PvPZT_ETRON: <p>Hi ! I tried to implement my first DQN agent for gym Cartpole, but it doesn’t seem to learn : the score at the end is worse than random play<br>
I tried some things :</p>
<ul>
<li>changing some parameters : learning rate, parameters for epsilon greedy, discount rate</li>
<li>changing the network architecture by making it much bigger</li>
<li>removing the target network<br>
Those don’t seem to work and I am very confused regarding what I’m doing wrong<br>
Thanks in advance for your help !</li>
</ul>
<aside class="onebox githubblob" data-onebox-src="https://github.com/baptisteeeeeeee/Cartpolerl/blob/main/cartpoleRL.py">
  <header class="source">

      <a href="https://github.com/baptisteeeeeeee/Cartpolerl/blob/main/cartpoleRL.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/baptisteeeeeeee/Cartpolerl/blob/main/cartpoleRL.py" target="_blank" rel="noopener nofollow ugc">baptisteeeeeeee/Cartpolerl/blob/main/cartpoleRL.py</a></h4>


      <pre><code class="lang-py">import gym
import math, random as rd, numpy as np, copy, matplotlib.pyplot as plt
import torch as T
import torch.nn as nn
import torch.functional as F
import torch.optim as omptim


env = gym.make('CartPole-v1', render_mode='human')
obs = env.reset()


gamma = 0.95
lr = 0.0001
epsilon, epmax, epmin, epdecay = 1, 1, 0.1, 0.005
N_episodes = 3000



n_input, n_hidden, n_out = 4, 5, 2
</code></pre>



  This file has been truncated. <a href="https://github.com/baptisteeeeeeee/Cartpolerl/blob/main/cartpoleRL.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


J_Johnson: <p>Do you have a chart of the progress? What I’ve found is DQNs often get better up to a point and then much worse if you keep training them. So it’s good to set milestones to save.</p>

J_Johnson: <p>Getting the correct rewards and Bellman’s target can often be a weak point and may need some tweaking. This developer was having a similar issue(albeit in Keras): <a href="https://adgefficiency.com/dqn-debugging/" class="inline-onebox" rel="noopener nofollow ugc">DQN debugging using Open AI gym Cartpole - ADG Efficiency</a></p>
<p>So you might need to review and tweak accordingly. DQNs are an ongoing area of research.</p>

J_Johnson: <p>Last comment, Pytorch has a tutorial with code you could give a try. It worked when I tried it at improving over time.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html</a></p>

vmoens: <p>Minor note here:<br>
We’re working on improving the DQN tutorial, you can check it there:</p><aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/tutorials/pull/2030">
  <header class="source">

      <a href="https://github.com/pytorch/tutorials/pull/2030" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/tutorials</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/tutorials/pull/2030" target="_blank" rel="noopener nofollow ugc">[WIP] Improve training of DQN tutorial</a>
      </h4>

    <div class="branches">
      <code>pytorch:master</code> ← <code>SiftingSands:DQN_revise_training</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2022-09-07" data-time="03:53:04" data-timezone="UTC">03:53AM - 07 Sep 22 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/SiftingSands" target="_blank" rel="noopener nofollow ugc">
            <img alt="SiftingSands" src="https://avatars.githubusercontent.com/u/43226539?v=4" class="onebox-avatar-inline" width="20" height="20">
            SiftingSands
          </a>
        </div>

        <div class="lines" title="28 commits changed 3 files with 92 additions and 148 deletions">
          <a href="https://github.com/pytorch/tutorials/pull/2030/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+92</span>
            <span class="removed">-148</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Following up the discussion from https://github.com/pytorch/tutorials/pull/2026
<span class="show-more-container"><a href="https://github.com/pytorch/tutorials/pull/2030" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">

I still need to do multiple runs to get a semblance of the statistics of # episodes vs duration for both the original and my changes. The slight increase in model capacity still only uses ~1.5 GB of VRAM, so it should be pretty accessible and training is still relatively quick.

Here's the reward history for one run of these tweaks when I was doing a bunch of trial and error (spent an embarrassing amount of time tweaking hyperparameters and rewards).

![duration_only_reward_03](https://user-images.githubusercontent.com/43226539/188784588-87a82d27-2f28-4e71-ae6e-1e4e5b3b8f72.png)

@vmoens Feel free to change (or completely discard) anything based on your findings. I haven't tried tweaking anything else in the training pipeline.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>