Don't understand why only Tensors of floating point dtype can require gradients

miraboreasu: <pre><code class="lang-auto">import torch
from torch.autograd.functional import jacobian

def get_f (x):      
       
    t1= x[0]**2
    t2= x[1]**3
    t3= x[2]**4
    f[0]= t1+t2+t3
    f[1]= 2*(t1+t2+t3)
    f[2]= 3*(t1+t2+t3)
    
    return f
    
x=torch.arange(0,3)
f=torch.arange(0,3)
J = jacobian(get_f, x).detach().numpy()

</code></pre>
<p>What I want J is a 3x3 matrix</p>
<pre><code class="lang-auto">df[0]/dx[0]   df[0]/dx[1]  df[0]/dx[2]
df[1]/dx[0]   df[1]/dx[1]  df[1]/dx[2]
df[2]/dx[0]   df[2]/dx[1]  df[2]/dx[2]
</code></pre>
<p>But I encounter the error<br>
RuntimeError: Only Tensors of floating point and complex dtype can require gradients<br>
I believe the requires_grad is True by default</p>
<p>Is there no way to diff the int32 (default) or I am doing something wrong?</p>

ptrblck: <p>I wouldn’t know how and if gradients are defined on “integer functions”. E.g. just take a simple example of <code>f(x) = x**2</code>.<br>
For floating point numbers you would see:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e282723d6bd3d33c4df9957df5cb6c1ff4028d97.png" alt="image" data-base62-sha1="wjNo3Wtcj2gasNOdP0VoaAUMtBJ" width="375" height="248"></p>
<p>and can draw the gradient directly into the plot.</p>
<p>But I guess this is how this simple function would look if only integers are used:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7ca3add6d30907ad27502e3678758132dd452379.png" alt="image" data-base62-sha1="hMBT01pO5l9YmfL0paV6TSs4mqR" width="368" height="248"></p>
<p>Would this mean that the gradient is everywhere 0 besides at the dots (where it would then be +/- Inf)?<br>
If so, then I don’t think it would make sense to allow Autograd to accept integer values.<br>
In case you are expecting integer outputs, it might be better to round the result.</p>
<p>Also, I’m sure other users such as <a class="mention" href="/u/kfrank">@KFrank</a>, <a class="mention" href="/u/tom">@tom</a>, and <a class="mention" href="/u/alband">@albanD</a> might have a better explanation.</p>

miraboreasu: <p>Thanks, another following question, maybe I am wrong, so the AD in PyTorch is doing finite difference? so you need it to be continuous? I am new to AD, a silly question, why can’t it perform dx**2/dx=2x</p>

AlphaBetaGamma96: <p>To comment on the <code>dx**2/dx = 2x</code> question, AD isn’t computing a symbolic expression for the gradient. It’s basically using the chain rule to calculate your derivative directly. If you want to read a paper on it, I’d recommend <a href="https://www.jmlr.org/papers/volume18/17-468/17-468.pdf" rel="noopener nofollow ugc">Automatic Differentiation in Machine Learning: a Survey</a> which will explain to you why AD isn’t symbolic  differentiation nor numerical differentiation</p>
<p>If you do want an expression you could use the <code>functorch</code> library, although that’s a bit more complicated than standard PyTorch. An example for your function would be something like this,</p>
<pre><code class="lang-auto">import torch
from functorch import grad, vmap

def f(x):
  return x**2

x = torch.arange(1,5,dtype=torch.float32) #dummy input
 
df_dx = grad(f) #creates explicit function via reverse-mode AD

vmap(df_dx, in_dims=(0))(x) #returns tensor([2., 4., 6., 8.])
</code></pre>

albanD: <p>The problem is not with how we compute the gradient but the definition of gradients itself.<br>
Mathematically, for gradients to be defined, you need a continuous function (at least locally).</p>

christophezei: <p>I totally agree but if I want to do inference with integers in this case will it be possible ?</p>

albanD: <p>That is possible indeed. Just make sure not to set requires_grad=True on these Tensor and autograd won’t run!</p>

christophezei: <p>Like I am saving my weights in a dict as integer but when I load them using .load_state_dict() function it doesn’t work they got converted to floats and if I try to convert them to int using .to(torch.uint8) on the model I have an error any suggestions?</p>