Creating a Clipped Loss Function

username: diegoalejogm
cooked: <p>Hi!</p>
<p>According to the  <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noopener">DeepMind DQL Paper</a>, the error term is clipped between -1 and 1. I am using clamp for that, but using it doesn’t allow me to use a default loss function like MSELoss.</p>
<p>How can I proceed to do this?</p>
<p>Code may be found in <a href="https://github.com/diegoalejogm/deep-q-learning/blob/master/Deep%20Q%20Learning.ipynb" rel="nofollow noopener">this repo</a>.</p>

username: SimonW
cooked: <p>I don’t understand. Why can’t you use error fns with clamp?</p>

username: diegoalejogm
cooked: <p>I don’t know how to do it correctly. If I try to use default errors, they usually take two arguments (target, prediction), but then I am unable to clamp on the loss for each pair. On the other hand, if I use clamp(target - prediction, min, max), I end up with only one tensor, and then I can’t use default errors.</p>
<p>Does it make sense?</p>

username: SimonW
cooked: <p>Oh I see. You can try the <code>reduce=False</code> kwarg on loss functions so they give you a tensor. Then you can do clamp and reduction yourself <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: diegoalejogm
cooked: <p>Hi <a class="mention" href="/u/simonw">@SimonW</a>, thanks for your help! I’ve just updated the optimizer:</p>
<pre><code>loss_func = torch.nn.MSELoss(size_average=False, reduce=False)
</code></pre>
<p>And also coded the backward pass accordingly:</p>
<pre><code># Run backward pass
error = loss_func(q_phi, y)
error = torch.clamp(error, min=-1, max=1)**2
error = error.sum() 
error.backward()
</code></pre>
<p>And it seems like no errors appear, which implies that the ‘backward’ operation is running correctly!</p>
<p>Will test it out in the Atari Environment and let you know how it goes. The code is in <a href="https://github.com/diegoalejogm/deep-q-learning/blob/master/Deep%20Q%20Learning.ipynb" rel="nofollow noopener">here</a> in case anyone wants to check it out meanwhile.</p>
<p>Thanks again!</p>

username: Hailin_Chen
cooked: <p>With code snippet below</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
w = Variable(torch.Tensor([1.]), requires_grad=True)
l = w**2
l_clip = l.clamp(max=0.8)
torch.autograd.grad(l, w, retain_graph=True) # prints (tensor([ 2.]),)
torch.autograd.grad(l_clip, w, retain_graph=True) # prints (tensor([ 0.]),)
</code></pre>
<p>It seems that clamp operation stops gradient flow if the value is clipped. Thus I think your code above might not work as you intended.</p>

username: diegoalejogm
cooked: <p><a class="mention" href="/u/hailin_chen">@Hailin_Chen</a>  Could you propose a solution for this issue?</p>

username: VihaanAkshaay
cooked: <p>Hey <a class="mention" href="/u/diegoalejogm">@diegoalejogm</a>, I was trying to implement the DQN algorithm by myself and had faced the same issue.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82c9c78b2b22bdf056907450c952b9e774e94e35.png" data-download-href="https://discuss.pytorch.org/uploads/default/82c9c78b2b22bdf056907450c952b9e774e94e35" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c9c78b2b22bdf056907450c952b9e774e94e35_2_517x117.png" alt="image" data-base62-sha1="iF0nF9qhLLdMi8AKa05IozrLvbT" width="517" height="117" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c9c78b2b22bdf056907450c952b9e774e94e35_2_517x117.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c9c78b2b22bdf056907450c952b9e774e94e35_2_775x175.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82c9c78b2b22bdf056907450c952b9e774e94e35_2_1034x234.png 2x" data-dominant-color="DFDFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1285×292 52.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I understand right, it is mentioned to clip the update of the difference, which probably means the gradient. so this becomes another case of Gradient Clipping and not the Loss clipping directly.</p>
<p>The official DQN code in the pytorch website does gradient clipping as well.<br>
You can find the code here - <a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="inline-onebox" rel="noopener nofollow ugc">Reinforcement Learning (DQN) Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<pre><code class="lang-auto">
#Optimize the model

optimizer.zero_grad()
loss.backward()
for param in policy_net.parameters():
    param,grad.data.clamp_(-1,1)
optimizer.step()

</code></pre>