Output of NN must be between -10 to 10

username: Pranavk
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>  I have a small NN with 4 outputs and I want that output to be in the range -10 to 10 strictly , I thought of using the tanh at the output and then multiplying it by 10. But I want to know if there is a better way to achieve this? because I found that the most values that I got by using tanh are very close to 0 like between range of -0.2 to 0.2 so after multiplying by 10 they become -2 to 2.</p>

username: AlphaBetaGamma96
cooked: <p>One thing you could do is, remove the <code>nn.Tanh()</code> and just have a <code>nn.Linear</code> module and then wrap the output with a <code>torch.clip</code> <a href="https://pytorch.org/docs/stable/generated/torch.clip.html" rel="noopener nofollow ugc">see here</a>. That could work!</p>

username: raharth
cooked: <p>I honestly think, that rescaling the target to [-1, 1] and using tanh-activation is the most solid way of approaching it.</p>
<p>I would probably not use clipping since you probably force the gradient to 0 if I’m not mistaken.</p>

username: AlphaBetaGamma96
cooked: <p>I think with clipping the gradient would only be zero for outputs out of the [-10,10] range. Rescaling to [-1,1] is a solution but you can have issues with outliers force most outputs near the extremes (i.e +/- 10); clipping was just a way to circumvent that!</p>

username: raharth
cooked: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="4" data-topic="120113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>I think with clipping the gradient would only be zero for outputs out of the [-10,10] range</p>
</blockquote>
</aside>
<p>Clipping gradients is not related to your output range I guess, it just prevents the gradients from exploding.</p>
<p>Clipping the output, will as you said mess with the gradients outside of the range you have defined, but would work perfectly fine within them. Though in my experience this will cause trouble especially with REINFORCE, but potentially with all algorithms.</p>
<aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="4" data-topic="120113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>can have issues with outliers force most outputs near the extremes (i.e +/- 10)</p>
</blockquote>
</aside>
<p>I’m not sure if I got you correctly, but what I meant was not just rescaling, but rescaling and using a sigmoid/tanh as output activation, then you have a constant gradient and do not have the issue of your values running “out of bounds”. Should have added that to the original post (in fact I will now). Or did I get you wrong?</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="raharth" data-post="5" data-topic="120113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raharth/48/14974_2.png" class="avatar"> raharth:</div>
<blockquote>
<p>Clipping gradients is not related to your output range I guess, it just prevents the gradients from exploding.</p>
</blockquote>
</aside>
<p>Yes, what I meant by them is, that you clip the output in the range of [-10,10] (not the gradients of the outputs). Although, clipping the output would in effect set the gradients to 0 for any output with |y| &gt; 10 (like you said).</p>
<aside class="quote no-group" data-username="raharth" data-post="5" data-topic="120113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raharth/48/14974_2.png" class="avatar"> raharth:</div>
<blockquote>
<p>I’m not sure if I got you correctly, but what I meant was not just rescaling, but rescaling and using a sigmoid/tanh as output activation, then you have a constant gradient and do not have the issue of your values running “out of bounds”.</p>
</blockquote>
</aside>
<p>What I mean is, if you re-scale the outputs you can have issues where outliers will affect the rescaling. For example, if most outputs are near say -0.5, and a few outputs are near 1. Then the rescaling will be distorted by the outliers near 1. One way to circumvent that is to clip the outliers and then rescale. Having a bounded activation function will mitigate this, but it can still be an issue. It’s just something to keep in mind when rescaling data!</p>