Find non zero elements in a tensor

username: wasiahmad
cooked: <p>I want to find the number of non-zero elements in a tensor along a particular axis. Is there any PyTorch function which can do this?</p>
<p>I tried to use the <a href="http://pytorch.org/docs/master/torch.html?highlight=nonzero#torch.nonzero" rel="nofollow noopener">nonzero()</a> method in PyTorch.</p>
<pre><code>torch.nonzero(losses).size(0)
</code></pre>
<p>Here, <code>lossess</code> is a tensor of shape <code>64 x 1</code>. When I run the above statement, it gives me the following error.</p>
<pre><code>TypeError: Type Variable doesn't implement stateless method nonzero
</code></pre>
<p>But if I run, <code>torch.nonzero(losses.data).size(0)</code>, then it works fine.</p>
<p>Any clue, why this is happening?</p>

username: alexis-jacq
cooked: <p>You can only call <code>torch.nonzero()</code> on a simple tensor, not a variable. It makes sens: I doubt that the counting of non-zero element would be differentiable.</p>
<p>but you have sum( abs( x/ (abs(x) + epsilon) )), that approximates the number of zero, and is differentiable.</p>

username: wasiahmad
cooked: <p>Actually I was trying to take the average of all non-zero elements in a 1-d tensor which is actually the total loss for my model. I am doing the following.</p>
<pre><code>loss = losses.sum() / torch.nonzero(losses.data).size(0)
</code></pre>
<p>It is working as expected and also backpropagation is not causing any problem, so I am assuming taking average of non-zero elements is differentiable. Do you have any thought about it?</p>

username: dpernes
cooked: <p>Yes, of course your total loss <em>L</em> is (piecewise) differentiable. It can be more formally defined as:</p>
<p><em>L</em> = sum( <em>Li</em> ) / sum( <em>1</em>{<em>Li</em> ≠ 0} ),</p>
<p>where <em>1</em>{<em>c</em>} is the indicator function (which is 1 when <em>c</em> is true and 0 otherwise).<br>
Clearly, the function <em>f</em> ( <em>Li</em> ) = <em>1</em>{<em>Li</em> ≠ 0} has derivative equal to 0 everywhere, except at <em>Li</em> = 0, where the derivative does not exist. In practice, you may assume that it is 0 everywhere.</p>
<p>Your biggest concern should be ensuring that you have no problem in using the tensor <code>losses.data</code> instead of the variable <code>losses</code>. This is because PyTorch will see <code>torch.nonzero(losses.data).size(0)</code> as a constant and not as a function of <code>losses</code>. Luckily, you may easily check that the derivative of <em>L</em> w.r.t. each of the losses <em>Lj</em> is the same whether you consider sum( <em>1</em>{<em>Li</em> ≠ 0} ) as a function of <em>Lj</em> or not:</p>
<p><em>dL</em> / <em>dLj</em> = 1 / sum( <em>1</em>{<em>Li</em> ≠ 0} )</p>

username: alexis-jacq
cooked: <p><a class="mention" href="/u/wasiahmad">@wasiahmad</a>, what you are minimizing is just <code>losses.sum()</code>, and your gradient descent steps are multiplied by a weight that depends on the number of non-zeros element, different at each iteration. But nothing guarantees that it will minimize sum(x)/non-zeros(x) for all x, which is (I think) what you want to do.</p>

username: fliman
cooked: <p>Just found, pytorch nonzero() is much slower than the numpy counterpart</p>

username: Tgaaly
cooked: <p>noticed this too! seems <code>nonzero()</code> is super slow. It also varies a lot every time its called. We had a situation where the first time its called it runs very fast and then subsequent calls run 10x slower.</p>

username: bx_p
cooked: <p>Have the same problem. For the same function with the exact input, first time runs 2.1s and second time 0.002s. Don’t know exactly what causes this problem… experience it in Pytorch 1.0</p>

username: Brando_Miranda
cooked: <p>perhaps a sigmoid could also work (since counting is like a step function and sigmoids are like step functions).</p>
<p>(I think u can even use a relu…)</p>