Add rows of Zeros with Mask

J_Johnson: <p>Suppose there are two torch tensors:</p>
<pre><code class="lang-auto">A=torch.arange(0,10)
B=torch.tensor([0,2,4,7,9])
</code></pre>
<p>First, I would like to produce a mask where B is missing values found in sequence A.<br>
Then I would like to apply that mask to another tensor of size:</p>
<p><code>C=torch.rand(5,2)</code></p>
<p>So that for every index missing in B, zeros are filled on that row of C. The final output should be of size (10,2).</p>
<p>I could do it with a loop, but the size of these tensors are in the millions of rows. Is there a parallelized method to approach this problem?</p>

Matias_Vasquez: <p>The way you described it, you can do something like this.</p>
<pre><code class="lang-python">A=torch.arange(0,10)
B=torch.tensor([0,2,4,7,9])

mask = torch.all(A - B.unsqueeze(0).T, dim=0)

C = torch.rand(5, 2)
D = torch.zeros(D.shape[0] + mask.count_nonzero(), 2)

D[mask.logical_not(), :] = C
print(C)
print(D)
</code></pre>
<pre><code class="lang-python"># Output:
# C
tensor([[0.8500, 0.1220],
        [0.8756, 0.0267],
        [0.6929, 0.1234],
        [0.4816, 0.5664],
        [0.0311, 0.8782]])
# D
tensor([[0.8500, 0.1220],
        [0.0000, 0.0000],
        [0.8756, 0.0267],
        [0.0000, 0.0000],
        [0.6929, 0.1234],
        [0.0000, 0.0000],
        [0.0000, 0.0000],
        [0.4816, 0.5664],
        [0.0000, 0.0000],
        [0.0311, 0.8782]])
</code></pre>
<p>But if you know the total size, and <code>B</code> has the row index where you want the information, then you can do something like this and get the same result.</p>
<pre><code class="lang-python">E = torch.zeros(10, 2)

E[B] = C
print(E)
</code></pre>

J_Johnson: <aside class="quote no-group" data-username="Matias_Vasquez" data-post="2" data-topic="150473">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar"> Matias_Vasquez:</div>
<blockquote>
<p><code>D = torch.zeros(D.shape[0] + mask.count_nonzero(), 2)</code></p>
</blockquote>
</aside>
<p>Perfect. But I just suggest you edit this line to:</p>
<p><code>D = torch.zeros(C.shape[0] + mask.count_nonzero(), 2)</code></p>

J_Johnson: <p>Seems that this line maxes out memory:</p>
<p><code>mask = torch.all(A - B.unsqueeze(0).T, dim=0)</code></p>
<p>Try with:</p>
<pre><code class="lang-auto">import torch

A=torch.arange(0,2000000)
B=torch.arange(0,1000000)
C=torch.rand(1000000,4)

mask = torch.all(A - B.unsqueeze(0).T, dim=0)
D = torch.zeros(C.shape[0] + mask.count_nonzero(), 4)

D[mask.logical_not(), :] = C
print(C)
print(D)
</code></pre>

Matias_Vasquez: <p>I would then suggest to use the other method.</p>
<pre><code class="lang-python">import torch

A=torch.arange(0,2000000)
B=torch.arange(0,1000000)
C=torch.rand(1000000,4)

D = torch.zeros(A.shape[0], 4)
D[B, :] = C

print(C)
print(D)
</code></pre>

J_Johnson: <p>Looks like I still need to figure out a way to get the indices, then. In the actual example, A can also have values missing.</p>
<p>I think what I’ll do, instead, is just upload the B data to the A table in Mysql and then pull it all in that way. Thanks.</p>