Why does this function break the gradient tree? :thinking:

spacemeerkat: <p>I have a function that is used at the end of my model and I would like to backpropogate the error through this function. However when I check the gradients they are extremely small leading me to believe the gradient tracking is not working as desired.</p>
<p>The function itself looks as follows:</p>
<pre><code class="lang-auto">z = torch.stack([*map(i.__eq__,torch.unique(i))]).type(torch.float)*(torch.stack([v]*z.shape[0])).type(torch.float)
</code></pre>
<p>It’s quite a complicated function (from <a href="https://discuss.pytorch.org/t/set-2d-tensor-elements-based-on-2d-index-tensor/60862">this original post</a>) which uses <code>*map</code> and <code>.__eq__</code> which are not native PyTorch functions and therefore I have a suspicion that one of these are the main issue with tracking gradients.</p>
<p>As the operation is quite complex, I have provided a graphical representation for visualising the operation below to help understand what the line does for given input tensors: <code>i</code>, <code>v</code>, and <code>z</code>.  Here, the tensor <code>i</code> dictates which channels of tensor <code>z</code> that the elements of tensor <code>v</code> fall into.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a90d5f2e699e9af083f63d840f20d3f18b90c88f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a90d5f2e699e9af083f63d840f20d3f18b90c88f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a90d5f2e699e9af083f63d840f20d3f18b90c88f_2_690x462.png" alt="image" data-base62-sha1="o7vlazlprLx55P7QMx1EsSgxOeH" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a90d5f2e699e9af083f63d840f20d3f18b90c88f_2_690x462.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a90d5f2e699e9af083f63d840f20d3f18b90c88f_2_1035x693.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a90d5f2e699e9af083f63d840f20d3f18b90c88f.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1048×703 50.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there a quick and easy way to test the gradient tracking through this function?</p>
<p>
Side note: If there is no way to quickly test whether the gradients can be tracked, maybe it is worth me posting a feature request to have an almost unit-test style functionality for checking functions are appropriate to place within networks…?
</p>

albanD: <p>Hi</p>
<p>I agree it’s very fun to write one liners like that, but that does not really make the code easily readable.</p>
<p>One thing to check is wether or not your output does require gradients or not. In this case, I think the left side of the multiplication won’t as the <code>__eq__</code> operation is not differentiable.<br>
The right side though should be differentiable. (as long as <code>v</code> is a Tensor that requires gradients).</p>
<p>For testing if the gradients computed are the actual gradients of your function, you can use our <a href="https://pytorch.org/docs/stable/autograd.html?highlight=gradcheck#torch.autograd.gradcheck">gradcheck</a> utility. You must run this in double precision. And it will check that gradients computed with finite difference match the ones from the autograd.</p>

spacemeerkat: <p>Thanks for replying!</p>
<p>I agree that was a little cheeky of me, I’ve tried to make it a bit clearer by providing a visual representation of the code’s operation. I tried to avoid repeating what I wrote for the post linked in my question as I’m more focused on wondering why the gradient tree breaks here but I realise I made the question difficult to understand because of that <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>I’m going to check whether the output of the operation requires gradients now and split up the code into the two parts of the multiplication like you suggest and check them both.</p>
<p>Thanks for the pointer to gradcheck I’m going to see if I can get that working too, it’d be nice to seed a testfile with checks to track the gradient flow!</p>

albanD: <p>Hi,</p>
<p>Given the image, scatter will be a better fit.<br>
I answered on your other post (sorry for not seeing it earlier).</p>
<p>The gradient for scatter are fairly simple:</p>
<ul>
<li>Gradient flow back to the scattered values</li>
<li>Gradients won’t flow back to the indices.</li>
</ul>

spacemeerkat: <p>Ahhh that’s great, it’s so simple and clean! <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=12" title=":open_mouth:" class="emoji" alt=":open_mouth:" loading="lazy" width="20" height="20"> Thank you for having a look! Out of curiosity is there a way to allow the gradient to flow back to the indices as well? In my case these are actually something that needs learning alongside the scattered values…it’s a strange network!</p>
<p>Think, something like this, where Index is <code>i</code> in the above visualisation, Source is <code>v</code>, and the output cube is <code>z</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ab49d32e4ddcc5c092715eb28e39e03df52ffff.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ab49d32e4ddcc5c092715eb28e39e03df52ffff" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2ab49d32e4ddcc5c092715eb28e39e03df52ffff_2_690x287.png" alt="image" data-base62-sha1="65N15CVD5VkeHd6hBQX9i6ltU63" width="690" height="287" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2ab49d32e4ddcc5c092715eb28e39e03df52ffff_2_690x287.png, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ab49d32e4ddcc5c092715eb28e39e03df52ffff.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ab49d32e4ddcc5c092715eb28e39e03df52ffff.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1007×419 29.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here, I think because the index is something that is learned through the CNN part of the network there should be some way to allow for the gradients to flow back through the Index tensor. (See <a href="https://discuss.pytorch.org/t/does-scatter--support-autograd/8587/3">related post</a> for this argument)</p>
<p>Edit: Perhaps using the out-of-place operation <code>torch.Tensor.scatter()</code> solves this?</p>

albanD: <p>The problem is that indices are integers. So gradients don’t really exist for them.<br>
For the same reason that argmax is not differentiable.</p>
<p>The trick that is often used to replace indexing is to use weighted sum.<br>
Instead of having an index in {0, 1, 2}, you have 3 values that sum to 1. And that gives you the probability of the index to be each of these values. Then you can replace indexing by weighted sum by these weights.<br>
This works in some cases where there are not too many indices. But might not be feasible if your usecase actually require indices.</p>

spacemeerkat: <p>Hi! Thanks for getting back to me!</p>
<p>I can give that a try with my usecase and see what happens! It might not work as in actual fact I have (1,120,64,64) cube inputs/outputs but it’s worth a shot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The difficulty here is that the network is specifically designed to learn the indices as a 2D tensor before blowing up to the full cube…I’m not sure how to get around this without indexing being incurred as a result (eventually) because the step from learned parameters to the index tensor is an analytical one rather than a learned one if that makes sense…</p>
<p>Perhaps there is a way of combining scatter with <code>grid_sample</code> like in <a href="https://discuss.pytorch.org/t/differentiable-indexing/17647/6">this post</a>? (Although I see in <a href="https://pytorch.org/docs/0.3.1/nn.html#torch.nn.functional.grid_sample" rel="nofollow noopener">the docs</a> that it only supports spatial indexing, so 2D idexing)</p>

spacemeerkat: <p>As the creation of the learned index tensor is a multi-stage process (involving the learning of a float tensor which is then transformed into a tensor of intigers), do you think there is any way to bypass the intiger-index tensor for calculating the gradient from inside the gradient tree? Perhaps a subunit of somekind?</p>
<p>Something like the following, where the green arrows show the gradient flow and the black arrows show the forward pass…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87231ccff3160e64382c55a37170c33e375660f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/87231ccff3160e64382c55a37170c33e375660f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87231ccff3160e64382c55a37170c33e375660f9_2_690x275.png" alt="image" data-base62-sha1="jhtGA3amfQeELsKxzz9XEr6Yw7D" width="690" height="275" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87231ccff3160e64382c55a37170c33e375660f9_2_690x275.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87231ccff3160e64382c55a37170c33e375660f9_2_1035x412.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/7/87231ccff3160e64382c55a37170c33e375660f9.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1351×540 52.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>Well I don’t know what this green arrow would compute.<br>
But if you have a formula for it, you can create a new autograd Function as described <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd">here</a> to define your own backward for a non-differentiable forward. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

spacemeerkat: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, I’ve been thinking about this reply a lot and I think I may have a way to do it but I thought I’d run it by you as initial tests have shown that the gradient tree is still broken somewhere.</p>
<p>So my crazy idea looks as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c1c008065865d91a1539a499a7420957d7c996b.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c1c008065865d91a1539a499a7420957d7c996b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c1c008065865d91a1539a499a7420957d7c996b_2_690x212.png" alt="image" data-base62-sha1="mh0tItcSVTjZ6pB5pmxsywiIwUr" width="690" height="212" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c1c008065865d91a1539a499a7420957d7c996b_2_690x212.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c1c008065865d91a1539a499a7420957d7c996b_2_1035x318.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c1c008065865d91a1539a499a7420957d7c996b_2_1380x424.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1512×465 49.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Where <code>I</code> and <code>S</code> are flattened replacements of the 2D arrays as seen in the <a href="https://discuss.pytorch.org/t/why-does-this-function-break-the-gradient-tree/63620/5">previous post</a>.</p>
<p>Now instead of indexing, the indices from <code>I</code> are used at the mean values of a series of normal distributions evaluated over the 2D array with dimensions N by x^2, and using some predefined σ.</p>
<p>Then a simple reshaping turns this 2D array back into the original cube shape.</p>
<p>As there is no indexing here, I shouldn’t think it would break the gradient tree…but is there something very obvious I’ve missed that you think would cause this method to fail?</p>
<hr>
<p><strong>Note</strong>: One can think of the 2D array (with dimensions N by x^2) as a flattened version of a cube where each row represents the channel indices of a cube spaxel C_ij. Pictorially this looks as follows:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/928b68eabe48a57dc2cb42f162ae0831827e0926.png" alt="image" data-base62-sha1="kUootTgsDOLiOEIN4hFSITrzWPI" width="255" height="143"></p>
<p>…where the maximum value of C_ij is normally distributed about I_ij with size S_ij.</p>

albanD: <p>Does that mean that you will need to backward through the operation of “evaluating a series of normal distributions” ? Is that differentiable wrt the mean?</p>

spacemeerkat: <p>I’ve done some thinking and I think it should be differentiable wrt the mean…</p>
<p>My normal is simply the following:</p>
<pre><code class="lang-auto">def normal(self,mu,std,auxiliary,amplitude):
        mu = mu.view(-1)
        amplitude = amplitude.view(-1)
        a = 1/(std*torch.sqrt(2*torch.tensor(pi)))
        b = torch.exp(-0.5*(((auxiliary-mu[:,None])/std)**2))
        return amplitude[:,None]*(a*b)
</code></pre>
<p>Where auxiliary just refers to the 2D array with dimensions N by x^2 above.</p>
<p>
Perhaps I should be using an inbuilt Pytorch normal distribution mechanism, or my handling of evaluating many normal distributions in one go (like in the code above) creates problems I can’t see… but as far as I can tell the differential of a normal distribution wrt the mean is possible. So I’m not sure what I’m doing wrong. 
</p>
<p>In retrospect I now understand that the differential of a series of evaluated normal distributions with means, given as <code>I</code> in the post above, will simply return a zero so this is not a good way of tackling the problem.</p>
<hr>
<p>That being said, would it not be possible to use an extension of autograd to simply backprop <code>dL/dI = tensor-of-ones</code> for the non-differentiable forward (in this case at the indexing step), or does that still prevent backprop functioning correctly for learning?</p>