Suspected error in DQN tutorial:

Liamtbo: <p>Hi everyone,</p>
<p>Quick disclaimer, I’m very new to torchrl, but after some thorough documentation reading, I believe I found some errors in the DQN torchrl tutorial <a href="https://pytorch.org/rl/stable/tutorials/getting-started-5.html" rel="noopener nofollow ugc">here</a></p>
<ol>
<li>The first error I saw was the creation of a egreedy module (placed in exploration_module), but not actually using it. Instead it looks like they used a workaround of 5000 random steps for exploration and then applying a greedy algorithm. What’s weird is they still step the egreedy module within the training loop which does nothing. One would of expected them to place the exploration_module as the policy in the collector, but instead they just passed the raw value network.</li>
<li>The second error, and far more significant, is the fast that that they didn’t set the DQNLoss parameter <code>delay_value = True</code>, which would create the target network (see <a href="https://pytorch.org/rl/stable/reference/generated/torchrl.objectives.DQNLoss.html" rel="noopener nofollow ugc">here</a>). I’m not sure what this means for when they step the softupdate.<br>
Please let me know if I made the error in my understanding, thanks.<br>
P.S. I haven’t submitted a PR on github because I’ve tried to correct for this in my own code and haven’t done so successfully yet.</li>
</ol>

vmoens: <p>Thanks for reporting!</p>
<p>I agree with (1) and we should use policy_explore in the collector.</p>
<p>For (2) it’s just a problem of doc, we changed the default and forgot to update the doc strings!</p>
<p>I will create issues in torchrl to track these.</p>

wilhelm: <p>Probably not the answer, that you would expect, but I can only suggest to drop <code>torchrl</code> and use <code>stable baselines3.</code> In that framework, you can develop your own implementation in PyTorch as well</p>

Liamtbo: <p>Thanks for the tip, is stable basline3 more industry standard?</p>

vmoens: <p>This PR fixed the doc and the built of the collector</p><aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/rl/pull/2429">
  <header class="source">

      <a href="https://github.com/pytorch/rl/pull/2429" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/rl</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/rl/pull/2429" target="_blank" rel="noopener nofollow ugc">[Doc] Fix policy in getting started</a>
      </h4>

    <div class="branches">
      <code>pytorch:main</code> ← <code>pytorch:correct-dqn-tuto</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2024-09-10" data-time="10:45:34" data-timezone="UTC">10:45AM - 10 Sep 24 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/vmoens" target="_blank" rel="noopener nofollow ugc">
            <img alt="vmoens" src="https://avatars.githubusercontent.com/u/25529882?v=4" class="onebox-avatar-inline" width="20" height="20">
            vmoens
          </a>
        </div>

        <div class="lines" title="1 commits changed 2 files with 2 additions and 2 deletions">
          <a href="https://github.com/pytorch/rl/pull/2429/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+2</span>
            <span class="removed">-2</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Fixes errors pointed in https://discuss.pytorch.org/t/suspected-error-in-dqn-tut<span class="show-more-container"><a href="https://github.com/pytorch/rl/pull/2429" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">orial/</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>RE TorchRL vs SB3: I’d say that torchrl is as robust as it gets, let’s not overstate what makes a library “industry-standard” or not based on a couple of typos in the doc/tutorials.</p>
<p>That said, SB3 is also a great library (although a totally different spirit of what TorchRL) and I’d definitly encourage you to check it out too!</p>
<p>A great example that comes to mind is how you build a buffer in both libs:<br>
In SB3 (from CleanRL):</p>
<pre data-code-wrap="python"><code class="lang-python">    rb = ReplayBuffer(
        args.buffer_size,
        envs.single_observation_space,
        envs.single_action_space,
        device,
        handle_timeout_termination=False,
    )
[...] later
rb.add(obs, real_next_obs, actions, rewards, terminations, infos)
[...]
data = rb.sample(args.batch_size)
data.observations
</code></pre>
<p>Here, data has fields <code>observations</code>, <code>actions</code> etc.<br>
You see that the position of the items in <code>add</code> determine their “role” in the buffer.<br>
Also, a big chunk of the buffer is based on numpy.<br>
It’s awesome if you “just” need a buffer that does exactly what this buffer does, but if you need a more refined usage (eg, you have multiple actions such as in MARL, your split between observations and info is not as clear cut, you want some fancy sampling…) it may not be super scalable.</p>
<p>TorchRL’s buffer is more generic and modular:</p>
<pre data-code-wrap="python"><code class="lang-python">rb = ReplayBuffer(storage=LazyTensorStorage(args.buffer_size, device=device))
[...]
transition = TensorDict(
            observations=obs,
            next_observations=real_next_obs,
            actions=torch.as_tensor(actions, device=device, dtype=torch.float),
            rewards=torch.as_tensor(rewards, device=device, dtype=torch.float),
            terminations=terminations,
            dones=terminations,
            batch_size=obs.shape[0],
            device=device
)
rb.extend(transition)
data = rb.sample(batch_size)
data["observations"]
</code></pre>
<p>You see that the buffer is more generic, we passed a storage on device (so our data stays on CUDA) and we could also ask for a sampler that has some extra features (samples slices of trajectories etc).</p>
<p>The philosophy of both libs is very different and it really depends whether you want to apply an existing algo to a specific problem that fits the requirements of the implementation (in which case SB3 is a good fit) or if you want to modify an existing implementation or do something that is a bit more off the charts.</p>