Slicing - Masking and gradient computation

username: mpalaourg
cooked: <p>Hello everyone,</p>
<p>I have a -nooby- question about gradient computation. Let’s say I have an input matrix <code>X</code> with shape of <code>[T, T]</code>. I want to process this matrix in a block diagonal way, where these blocks will have a shape of <code>[B, B]</code> with (<code>B &lt;&lt; T</code>).<br>
For the first implementation and to check if my logic make sense, I went with a simple for loop and in each step modify only one block of <code>X</code>  (add a dependency factor) and nothing else. The grad_fn attribute shows <code>(&lt;CopySlices object at 0x7f76c424a070&gt;, 0)</code> and I assume that the backpropagation is working as intended.<br>
This was extremely slow, so for the second implementation I went with a mask approach. My mask is known for a given <code>Χ</code>, so I precomputed it. I compute the dependency factor for each item in the matrix (<code>[Τ, Τ]</code>), but I want to keep only the masked factor (to be added in a block diagonal way to <code>Χ</code>). Finally, I compute <code>X = X + (mask * factor)</code>). This approach give <code>(&lt;MulBackward0 object at 0x7f76c4283be0&gt;, 0))</code>.</p>
<p>My question is: The two results I get from those implementations is slightly different (max absolute error <code>2e-5</code>). Is that because of numeric reasons? OR Is that because on the second approach I impact every item on the <code>[T, T]</code> matrix, even if outside the blocks the dependency factor I add is zero?</p>
<p>Thanks in advance for any help and insight you can provide!</p>

username: KFrank
cooked: <p>Hi George!</p>
<aside class="quote no-group" data-username="mpalaourg" data-post="1" data-topic="135832" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mpalaourg/48/43083_2.png" class="avatar"> mpalaourg:</div>
<blockquote>
<p>The two results I get from those implementations is slightly different (max absolute error <code>2e-5</code>). Is that because of numeric reasons?</p>
</blockquote>
</aside>
<p>I believe that your discrepancy is due to (accumulated) round-off<br>
error.</p>
<p>An easy way to test this is to repeat your computation using double<br>
precision (by using <code>torch.double</code> tensors).  If the discrepancy is due<br>
to round-off error it should drop dramatically, say to about 1e-13.  If the<br>
discrepancy were to remain around 2e-5, then it would be due to the<br>
two computations not being mathematically equivalent (or to some bug<br>
somewhere).</p>
<aside class="quote no-group">
<blockquote>
<p>OR Is that because on the second approach I impact every item on the <code>[T, T]</code> matrix, even if outside the blocks the dependency factor I add is zero?</p>
</blockquote>
</aside>
<p>Based on your description, I believe that your two computations should<br>
be mathematically equivalent (but without full details, I can’t really say<br>
for sure).</p>
<p>As an aside, do you ever work with the off-block-diagonal parts of your<br>
tensor <code>X</code>?  If not, you might consider creating, storing, and processing<br>
only the <code>[B, B]</code>-shaped blocks themselves.  Given that <code>B &lt;&lt; T</code>, doing<br>
so could result in significant time and memory savings.</p>
<p>Best.</p>
<p>K. Frank</p>

username: mpalaourg
cooked: <p>Hi Frank,<br>
Firstly thank you for your answer. You validate to me, that it was indeed a bug and not something I didn’t understand about autograd!</p>
<blockquote>
<p>An easy way to test this is to repeat your computation using double<br>
precision (by using torch.double tensors). If the discrepancy is due<br>
to round-off error it should drop dramatically, say to about 1e-13. If the<br>
discrepancy were to remain around 2e-5, then it would be due to the<br>
two computations not being mathematically equivalent (or to some bug<br>
somewhere).</p>
</blockquote>
<p>As I said before the behavior I documented it was because of a nasty little bug. In my tries to make it faster I found an error in my logic (broadcasting is to blame… or me <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">). After I fixed that, the error drop at <code>1e-10</code>, even without double tensors!</p>
<blockquote>
<p>As an aside, do you ever work with the off-block-diagonal parts of your<br>
tensor <code>X</code> ? If not, you might consider creating, storing, and processing<br>
only the <code>[B, B]</code> -shaped blocks themselves. Given that <code>B &lt;&lt; T</code> , doing<br>
so could result in significant time and memory savings.</p>
</blockquote>
<p>At least for now, I want all the <code>[T, T]</code> matrix with a bit processing in those blocks for the diagonal. If the logic of the architecture change I would definitely do that to improve my  memory usage. The time savings is already more than enough, after I get rid of that slow for loop!</p>
<p>PS. Thanks again for your help. I’ll mark your answer as the solution, because it was indeed a bug!</p>