Logits for only certain outputs creates infs

ZimoNitrome: <p>I have a network with a linear head followed by a sigmoid activation function. Most of my output neurons should be in a range 0-1 and thus sigmoid makes sense, but some of the neurons should output normalized (standardized) colors.</p>
<p>So for the last three neurons I do <code>torch.logit(x)</code> as they can be in the range [-1.3, 1.4]. I still encounter errors sometimes because some of the color neurons output 1.0, and <code>torch.logit(1.0)</code> gives <code>inf</code>.</p>
<p>Should I use something else other than <code>torch.logit()</code> after my activation functions for these special neurons? Maybe normalize (not standardize!) such that the range [0, 1] becomes [-1.3, 1.4]?</p>

KFrank: <p>Hi Zimo!</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="1" data-topic="135819" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>some of the color neurons output 1.0, and <code>torch.logit(1.0)</code> gives <code>inf</code>.</p>
</blockquote>
</aside>
<p>Instead of applying <code>sigmoid()</code> to all of your neurons and then applying<br>
<code>logit()</code> to your color neurons to undo the <code>sigmoid()</code>, apply <code>sigmoid()</code><br>
only to your non-color neurons so that the color neurons never get<br>
transformed.</p>
<p>(Applying <code>sigmoid()</code> to a large-enough (non-<code>inf</code>) value will map it<br>
to <code>1.0</code>, causing some loss of information, and you can’t undo the<br>
numerical “damage” by applying <code>logit()</code>, even though <em>mathematically</em><br>
<code>logit()</code> is the inverse function of <code>sigmoid()</code>.)</p>
<p>As an aside, you should consider whether you really want to apply<br>
<code>sigmoid()</code> to any of your neurons.  What subsequent processing<br>
do you do to your non-color neurons, and could you perform that<br>
processing in “log space,” that is, working with the logits that are<br>
output by your <code>Linear</code> head?  Doing so might prove more numerically<br>
stable.</p>
<p>Best.</p>
<p>K. Frank</p>

ZimoNitrome: <p>Thanks KFrank, great answer!</p>
<p>A lot of the other neurons correspond to 3d transformations of objects. I will describe down below how I currently re-scale the outputs:</p>
<ul>
<li>
<p>XYZ position in a restricted 1x1x1 box. Constrained by sigmoid.</p>
</li>
<li>
<p>XYZ rotation. Currently constrained by sigmod, then normalized to [0, 2*pi].</p>
</li>
<li>
<p>Width, height, depth. Constrained by sigmoid + epsilon since I desire a smallest possible size and 1x1x1 is the largest possible size in the restricted box.</p>
</li>
</ul>
<p>This is still a project in its early stage, so I might add some more outputs. Maybe ones that are restricted to something like [-5, 200]. How should I rescale logits to conform to these values? Maybe I shouldn’t and instead try to get the model to do that fitting for me?</p>
<p>The problem has been that 3D objects that fall outside of the 1x1x1 space (as in XYZ position) or objects that get a width/height of 0 get an inf/nan gradient.</p>