Does TorchRL CQL supports discrete action?

Deep_Gravity: <p>Hi all,</p>
<p>I wonder if anyone knows that TorchRL CQL supports discrete action? I’ve check the documentation, but it is not clear if it supports or not.</p>
<p>Thanks!</p>

vmoens: <p>Hi <a class="mention" href="/u/deep_gravity">@Deep_Gravity</a><br>
thanks for the question.<br>
We use <code>dist.rsample()</code> which works with continuous values only (except if you’re using tricks to reparametrize discrete values).<br>
This is something we can defo work on for the next version, as the paper tests the results on discrete settings too.<br>
<span class="mention">@BY571</span> any thoughts?</p>

BY571: <p>Hey <a class="mention" href="/u/deep_gravity">@Deep_Gravity</a>,</p>
<p>currently, TorchRL does not support the discrete (DQN-) CQL version. However, implementing it is straightforward, we might have to check how to make it work with the current continuous version or create a standalone discrete CQL loss like the discrete SAC <a class="mention" href="/u/vmoens">@vmoens</a> . I will give it a shot this week, <a class="mention" href="/u/deep_gravity">@Deep_Gravity</a> if you want you can follow the progress here <a href="https://github.com/pytorch/rl/pull/1666" class="inline-onebox" rel="noopener nofollow ugc">[WIP] Discrete CQL by BY571 · Pull Request #1666 · pytorch/rl · GitHub</a>. It might not get merged quickly but once the objective is done you could probably run it from that branch if you need to.</p>

BY571: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6d8d0f59b7f76d5d63528a72e70e2e66f10b253.png" data-download-href="https://discuss.pytorch.org/uploads/default/e6d8d0f59b7f76d5d63528a72e70e2e66f10b253" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6d8d0f59b7f76d5d63528a72e70e2e66f10b253_2_690x411.png" alt="image" data-base62-sha1="wWalnGxP1sT9MFriLcs8Bmx4jiX" width="690" height="411" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6d8d0f59b7f76d5d63528a72e70e2e66f10b253_2_690x411.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6d8d0f59b7f76d5d63528a72e70e2e66f10b253_2_1035x616.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6d8d0f59b7f76d5d63528a72e70e2e66f10b253.png 2x" data-dominant-color="252426"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1142×681 78.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a class="mention" href="/u/deep_gravity">@Deep_Gravity</a> I added the loss (+ example for cartpole) in the <a href="https://github.com/pytorch/rl/pull/1666" rel="noopener nofollow ugc">PR</a> and it works already. So in case you need to you can run experiments on that branch if you dont want to wait until the merge.</p>

Deep_Gravity: <p>Dear <a class="mention" href="/u/by571">@BY571</a>, thank you very much. I will look into it this week and might have some questions for which I’ll come back to you again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>