When is autograd available and is it always accurate?

username: Valiox
cooked: <p>I’ve been trying to wrap my head around autograd for a while now. From what I’ve learned about automatic differentiation in class, it is possible to compute the gradient of functions written with code even when doing overwrites, for loops, etc… Additionally, <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">the autograd tutorial</a> states:</p>
<blockquote>
<p>The  <code>autograd</code>  package provides automatic differentiation for all operations on Tensors.</p>
</blockquote>
<p>However, I understand that not everything is differentiable (e.g. a black-box function).</p>
<p>I’m currently reimplementing in PyTorch a neural network with a layer that performs a lookup operation in a table with an index derived from a prediction. That is, some floating-point input is rounded and cast as an integer in order to be used as an array index. Here is a mockup of what could be the forward pass of this layer:</p>
<pre><code class="lang-auto">    class LookupLayer(Module):
        def __init__(self):
            super().__init__()
            self.lookup_array = torch.arange(10)
            
        def forward(self, x):
            index = torch.round(x).int()
            index = torch.max(0, torch.min(len(self.lookup_array) - 1, index))
            return self.lookup_array[index]
</code></pre>
<p>The grad attribute of the parameters of the model for all layers above this one is <code>None</code>, which I assume is due to autograd being unable to compute the derivative of this layer. That makes sense to me, as I don’t think that <code>int()</code> has a derivative.</p>
<p>The author of the paper I am reproducing provides the backward pass implementation of the layer and so I believe that I should now put my layer as a subclass of <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc"><code>torch.autograd.Function</code></a>  rather than <code>torch.nn.Module</code> and reimplement the backward pass there myself. Please correct me if I am wrong.</p>
<p>My questions are:</p>
<ul>
<li>
<p>Is it trivial to tell if autograd is going to be able to derive the function or not?</p>
</li>
<li>
<p>If the forward pass is correct and if autograd is able to compute a gradient, is this gradient always correct?</p>
</li>
</ul>
<p>I’ve watched <a href="http://videolectures.net/deeplearning2017_johnson_automatic_differentiation/" rel="noopener nofollow ugc">this lecture</a> to try to find answers to these questions, but I could not quite understand everything. If you have more resources I would be happy to have them.</p>

username: rasbt
cooked: <blockquote>
<p>That is, some floating-point input is rounded and cast as an integer in order to be used as an array index</p>
</blockquote>
<p>I think the resulting gradient is 0 in that case. I think even if you would do something less “excotic” like predicting some output rounding the logits, and then use it to compute the MSE against a target label, it would give you a gradient of 0.</p>
<blockquote>
<p>From what I’ve learned about automatic differentiation in class, it is possible to compute the gradient of functions written with code even when doing overwrites</p>
</blockquote>
<p>I am by no means an expert in how autodiff is implemented, but I think I heard somewhere that everything in a computer should be differentiable because you can decompose everything into atomic operations (basically additions). But I am not sure that it’s implemented like that in the autodiff submodule. I have actually no idea how it’s implemented in PyTorch’s submodule, but if I had to speculate, based on what I’ve seen, it’s really manually defined for certain operations.</p>

username: albanD
cooked: <p>Hi,</p>
<p>For your first question, no there is no way at the moment. But it is quite “obvious” which function won’t give you usefull gradients. In particular functions that are piecewise constant (rounding, indexing etc) will give an almost always zero gradient and so won’t be useful.</p>
<p>For your second question: the autograd engine is based on the chain rule:</p>
<p>If you have f and g two differentiable functions, and you do <code>y=f(x)</code> and <code>z=g(y)</code>, then you have <code>dx/dz = dx/dy * dy/dz</code>.</p>
<p>You can apply this multiple times until each derivative is only computed for an “elementary” function.<br>
Then you simply need to implement the derivatives of these elementary functions and you get the derivative for the whole succession of functions.</p>
<p>So the gradients computed by pytorch will alway be correct as long as the condition above “f and g are differentiable” is valid. If they are not, it can break (or not break) in many unexpected ways and hells ensues <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_imp.png?v=6" title=":smiling_imp:" class="emoji" alt=":smiling_imp:"><br>
In practice it should behave properly for these cases as well but you can find single points where the computed gradient will be wrong (and we cannot detect that automatically…)</p>

username: cherepanovic
cooked: <aside class="quote no-group" data-username="Valiox" data-post="1" data-topic="35840">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valiox/48/19212_2.png" class="avatar"> Valiox:</div>
<blockquote>
<p>The <code>autograd</code> package provides automatic differentiation for all operations on Tensors.</p>
</blockquote>
</aside>
<p>Do I understood it correctly, that even if I didn’t specified a backward function the autograd would be able to provide derivates w.r.t. inputs of that function?</p>