Is there any soft way of counting positive values with grad reserved?

Ximeng: <p>Hi,</p>
<p>I need a differentiable metric to roughly reflect the level of positive values in a tensor, so literally counting is not necessary.</p>
<p>I tried two hard operations:</p>
<ol>
<li>relu+torch.sign+sum</li>
<li>torch.count_nonzero</li>
</ol>
<p>Result:</p>
<ol>
<li>requires_grad=True. But input.grad are all zeros after using “torch.sign” .</li>
<li>requires_grad=False</li>
</ol>
<p>I’m not sure if fc layer+relu+soft sign+sum is a potential option.</p>
<p>Any suggestions?</p>
<p>Thank you!</p>

KFrank: <p>Hi Ximeng!</p>
<aside class="quote no-group" data-username="Ximeng" data-post="1" data-topic="158975" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/8c91f0/48.png" class="avatar"> Ximeng:</div>
<blockquote>
<p>I need a differentiable metric to roughly reflect the level of positive values in a tensor,</p>
</blockquote>
</aside>
<p><code>sigmoid (x)</code>, which is differentiable, moves “softly” from <code>0</code> to <code>1</code> as <code>x</code><br>
moves from negative to positive.</p>
<p>You may shift where the transition occurs, <code>sigmoid (x - shift)</code>, and<br>
sharpen the transition, <code>sigmoid (sharpness * x)</code>.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; x = torch.randn (5, 8, requires_grad = True)
&gt;&gt;&gt; x
tensor([[-0.9788, -1.5154, -0.8222,  0.1214,  0.0716, -0.0872, -0.0253, -1.6267],
        [ 0.2230, -1.6746, -1.4725,  0.9721, -0.2191, -0.9397, -1.7756, -0.6259],
        [-1.1104,  1.1890,  1.3730,  0.4915,  0.3579, -0.1685, -0.8579, -1.0574],
        [ 0.2105,  1.9045,  1.8237,  1.5122, -0.3140, -0.0810, -1.3631, -0.0701],
        [-1.1876, -1.0787,  0.9551, -0.2958,  1.0663, -0.5134, -0.3846, -1.1481]],
       requires_grad=True)
&gt;&gt;&gt; (x &gt; 0).sum()
tensor(14)
&gt;&gt;&gt; torch.sigmoid (x)
tensor([[0.2731, 0.1801, 0.3053, 0.5303, 0.5179, 0.4782, 0.4937, 0.1643],
        [0.5555, 0.1578, 0.1866, 0.7255, 0.4454, 0.2810, 0.1448, 0.3484],
        [0.2478, 0.7666, 0.7979, 0.6204, 0.5885, 0.4580, 0.2978, 0.2578],
        [0.5524, 0.8704, 0.8610, 0.8194, 0.4221, 0.4798, 0.2037, 0.4825],
        [0.2337, 0.2538, 0.7221, 0.4266, 0.7439, 0.3744, 0.4050, 0.2408]],
       grad_fn=&lt;SigmoidBackward0&gt;)
&gt;&gt;&gt; torch.sigmoid (x).sum()
tensor(17.9145, grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; torch.sigmoid (10 * x).sum()
tensor(14.9508, grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; torch.sigmoid (100 * x).sum()
tensor(14.0741, grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Ximeng: <p>Thanks Frank. But I am not sure if the sigmoid manner will possibly cause a gradient vanishment problem.</p>
<p>c = torch.randn(10, requires_grad=True)<br>
d = torch.sigmoid (1000 * c).sum()<br>
d.backward()<br>
c.grad<br>
tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,<br>
1.6938e-22, 0.0000e+00, 0.0000e+00, 0.0000e+00])</p>
<p>So I tried another possible solution which might slightly mitigate the problem.<br>
a = torch.randn(10, requires_grad=True)<br>
b = torch.relu(a/(1e-3 + torch.abs(a))).sum()<br>
b.backward()<br>
a.grad<br>
tensor([0.0000, 0.0002, 0.0003, 0.0027, 0.0000, 0.0000, 0.0000, 0.0000, 0.0012,<br>
0.0029])</p>
<p>BTW, I noticed that gradients are also tiny for a normal softmax operation.<br>
a = torch.randn(8, requires_grad=True)<br>
b = torch.softmax(a, dim=0) .sum()<br>
b.backward()<br>
a.grad<br>
tensor([7.9883e-09, 1.6591e-08, 4.4367e-09, 1.7502e-08, 1.7472e-08, 9.1981e-09,<br>
2.4514e-09, 4.3570e-08])<br>
Obviously, softmax should always work(eg: in self-attention block) even if Float32 has about 7 significant digits if I remember correctly.  So I guess there must be some misunderstanding about the gradient above <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi Ximeng!</p>
<aside class="quote no-group quote-modified" data-username="Ximeng" data-post="3" data-topic="158975" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/8c91f0/48.png" class="avatar"> Ximeng:</div>
<blockquote>
<p>But I am not sure if the sigmoid manner will possibly cause a gradient vanishment problem.</p>
<p>…</p>
<p>d = torch.sigmoid (1000 * c).sum()</p>
</blockquote>
</aside>
<p><code>1000</code> is <em>very</em> large for the multiplier used to “sharpen” the <code>sigmoid()</code>.<br>
This causes the <code>sigmoid()</code> to become quite close to a (discontinuous)<br>
step function for which the gradients would be exactly zero.  Very small<br>
gradients (that underflow to zero) are to be expected here.</p>
<p>If you want “soft” counting, your “soft count” will be a floating-point number<br>
that only approximates your actual count (and you can get useful gradients).</p>
<p>If you want your “soft count” to very closely approximate the true count,<br>
its gradients will become very close to zero.  That’s the unavoidable<br>
trade-off.</p>
<aside class="quote no-group">
<blockquote>
<p>BTW, I noticed that gradients are also tiny for a normal softmax operation.<br>
a = torch.randn(8, requires_grad=True)<br>
b = torch.softmax(a, dim=0) .sum()<br>
b.backward()<br>
a.grad<br>
tensor([7.9883e-09, 1.6591e-08, 4.4367e-09, 1.7502e-08, 1.7472e-08, 9.1981e-09,<br>
2.4514e-09, 4.3570e-08])</p>
</blockquote>
</aside>
<p>The zero gradients (up to round-off error) are due to the fact that you<br>
used <code>sum()</code> to reduce the result of <code>softmax()</code> to a scalar on which you<br>
could call <code>.backward()</code>.  By definition, the <code>sum()</code> of <code>softmax()</code> is exactly<br>
one (up to round-off error), which is a constant, so the gradient of<br>
<code>softmax().sum()</code> is indeed zero.</p>
<p>You could try, for comparison, <code>b = torch.softmax(a, dim=0).exp().sum()</code><br>
and you will see that you get non-zero gradients.</p>
<p>Best.</p>
<p>K. Frank</p>

Ximeng: <p>Oh, I see. Thank you very much!</p>