Filter Out Undesired Rows

Marina_Drygala: <p>I am generating artifical data. I would like to filter out the rows of my input tensor that don’t satisfy a certain condition and then save the indices so that I can remove the corresponding rows from my output tensor.</p>

ptrblck: <p>Would it be possible to use your condition to index the tensor?<br>
Here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(10, 2)
condition = x &gt; 0.
row_cond = condition.all(1)
x[row_cond, :]
</code></pre>

Marina_Drygala: <p>I have a function that computes a value using all the inputs in a row, and I would like to filter out the rows that don’t meet a certain threshold for that value.</p>

Marina_Drygala: <p>Let me know if you want me to be more specific <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Could you use this calculated value to index your tensor?<br>
Let’s say your function computes the sum of each row:</p>
<pre><code class="lang-python">value = x.sum(1)
x[value&gt;threshold]
</code></pre>
<p>If that doesn’t work, could you post the shape of your tensor and the value you’ve calculated?</p>

Marina_Drygala: <p>A small example of what I am doing, with 5 training examples:</p>
<p>Input:<br>
tensor([[ 0.0166, -0.2023, -0.2503, -0.3227, -0.2823,  0.8440],<br>
[ 0.4075,  0.0052, -0.7873, -0.3248,  0.1329,  0.3014],<br>
[ 0.2826,  0.4441,  0.2709,  0.4514,  0.2911,  0.6008],<br>
[-0.1225,  0.0034, -0.2977,  0.3847,  0.5563,  0.6625],<br>
[-0.3808, -0.5172,  0.4302, -0.2792,  0.1753,  0.5419]]<br>
Output:<br>
tensor([[ 0.2294, -0.2380,  0.2742, -0.0511,  0.4272,  0.2381, -0.1149, -0.8085,<br>
0.2283, -0.8853,  0.1314,  0.0665, -0.2199,  0.8177,  0.0667,  0.4147],<br>
[ 0.4232, -0.5899, -0.3844,  0.9617, -0.9795, -0.0679, -0.0792,  0.7093,<br>
-0.0951,  0.2633, -0.0480, -0.5599, -0.5668, -0.4858, -0.9084, -0.6490],<br>
[ 0.2353,  0.6581,  0.0493, -0.4584,  0.4395, -0.3839, -0.2215, -0.5482,<br>
-0.3140, -0.9266,  0.4267,  0.3888,  0.1986,  0.4910,  0.4238,  0.0442],<br>
[ 0.1059,  0.0764,  0.5336,  0.6717,  0.7181,  0.5796, -0.2438, -0.0445,<br>
-0.2032,  0.5817,  0.1111,  0.9255,  0.5072, -0.8547,  0.2925,  0.9609],<br>
[ 0.8882, -0.0157,  0.3318, -0.9381, -0.3188,  0.4876, -0.9110,  0.8712,<br>
-0.6576,  0.3162, -0.0379,  0.1762,  0.0969, -0.9348, -0.2148, -0.6321]])</p>
<p>I have a function theta that takes a row of my input tensor and outputs a value.</p>
<p>if I apply theta to each row of my input I get:<br>
tensor([0.2515, 0.2275, 0.2988, 0.2581, 0.2819])</p>
<p>Now if I select 0.26 as my threshold value I would like to filter out rows 0,1,3 from my input and output tensors and be left with just 2 and 4.</p>

ptrblck: <p>Thanks for the example.<br>
Let’s call the first tensor <code>input</code>, the second <code>output</code> and the last <code>theta</code>.<br>
Would the following work:</p>
<pre><code class="lang-python">threshold = 0.26
idx = theta &gt; threshold
input_filt = input[idx]
output_filt = output[idx]
</code></pre>
<p>If you want to keep the negation, you could use:</p>
<pre><code class="lang-python">input_filt = input[~idx]
output_filt = output[~idx]
</code></pre>
<p>or just flip the <code>&gt;</code> sign.</p>
<p>Let me know, if that’s what you want.</p>

J_Johnson: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> - This is a very elegant solution. Thank you. Is there a way to apply this in the case where the mask should remove values that are equal to the previous value?</p>
<p>For example, suppose a tensor has:<br>
tensor([<br>
[0, 2.4],<br>
[0, -4.1],<br>
[1, 0.5],<br>
[0, 1.6]<br>
])<br>
And so the mask should only keep the last two values since in the first column they are not equal to the previous value.<br>
tensor([<br>
[1, 0.5],<br>
[0, 1.6]<br>
])</p>
<p>How would you suggest applying a mask in this case?</p>

J_Johnson: <p>Figured it out. Pretty simple actually.</p>
<pre><code class="lang-auto">condition = x[1:, 0:1] != x[:-1, 0:1]
row_cond=condition.all(1)
y=x[1:,:]

y=y[row_cond,:]
</code></pre>
<p>Cheers!</p>