Is the type conversion differentiable?

oahz48: <p>I am using a mask based on comparison:</p>
<pre><code class="lang-auto">def combine_images(target, x, y):
    diff1 = torch.abs(target - x)
    diff2 = torch.abs(target - y)
    mask = (diff1 &lt; diff2).float()
    target_new = x * mask + y * (1 - mask)
    return target_new
</code></pre>
<p>My question is that is this function differentiable since it includes a type conversion <code>.float()</code>? Thanks.</p>

MariosOreo: <p>Hi,</p>
<p>I just have a try on it with random tensors, it works.<br>
In all these tensors, only <strong>mask</strong> with <code>requires_grad=False</code>.</p>

oahz48: <p>Thanks. I have tried with</p>
<pre><code class="lang-auto">x = torch.tensor([0.0, 1.0], requires_grad=True)
y = torch.tensor([0.5, 0.5], requires_grad=True) 
mask = (x &lt; y).float() 
z = (x * mask + y * (1 - mask) * 10).sum()
z.backward()

In [18]: x.grad                                                                 
Out[18]: tensor([1., 0.])

In [19]: y.grad                                                                 
Out[19]: tensor([ 0., 10.])
</code></pre>
<p>It seems it is working and the result looks reasonable for this trivial case. Not sure if it applies to general cases.</p>

ptrblck: <p>Type conversions are “differentiable” as can be seen in this dummy mixed-precision example:</p>
<pre><code class="lang-python">x = torch.randn(1, 10, dtype=torch.float16, device='cuda')
w1 = torch.randn(10, 1, requires_grad=True, dtype=torch.float16, device='cuda')
w2 = torch.randn(1, 1, requires_grad=True, dtype=torch.float32, device='cuda')

output = torch.matmul(x, w1)
output = output.float()
output = torch.matmul(output, w2)

loss = (output - torch.randn(1, 1, dtype=torch.float32, device='cuda'))**2
loss.backward()

print(w1.grad)
print(w2.grad)
</code></pre>
<p>As you can see, I’m starting with <code>float16</code> input and parameters, and convert them to <code>float32</code> later, which just works fine, as Autograd just transforms the gradients back to the appropriate type.</p>

trinity: <p>For <code>long</code> it seems like the operation does not have a <code>grad_fn</code>. So is there any way to still use Autograd when the type conversion is to <code>long</code>?</p>

tom: <p>No, integral tensors cannot have gradient, only floating-point tensors can.<br>
The underlying reason is that integral-valued functions are not mathematically meaningfully differentiable (ie they might be (locally) constant, but that’s it).</p>

trinity: <p>Right, I see! Thanks for the clarification.</p>
<p>Slightly off-topic question then - inside a training loss, I need to access the values of a tensor [<code>y_true</code>] by indices. The other tensor [<code>y_pred</code>] which consists of the indices, is of type <code>float</code> and has <code>float</code> values. Since I need to compute the gradient, is there any way to access values of <code>y_true</code>, without rounding <code>y_pred</code> (would like to avoid this due to its zero gradient output almost everywhere) and then doing the type conversion of it to <code>long</code>? Please note that it is not possible to do any interpolation in <code>y_true</code> in this context. A minimal example of the said loss function is as following -</p>
<pre><code class="lang-auto">def trainloss(y_pred, y_true): 
    #y_pred is of shape [100,2], y_true is of shape [64,64]
    idx = torch.round(y_pred)  
    idx = idx.long()
    loss = y_true[idx[:,0],idx[:,1]]
    loss = torch.max(loss)
return loss
</code></pre>

tom: <p>We would likely want to think about what the derivative should be mathematically before we look at the implementation. What should happen to y_pred relative to loss? You write that no interpolation is possible, but so if y_pred had some gradient that causes the rounding to decrement idx by one from one step to the other, then you end up with <code>y_true[old_idx[:, 0] - 1, old_idx[:, 1]</code>. This would suggest that if that is smaller than that <code>y_pred[:, 0]</code> should have a positive gradient. (Leaving aside the batch thing…)</p>
<p>One variant where you try to avoid needing to differentiating y_pred is the REINFORCE algorithm in RL. Essentially, when y_pred is a sample from a probability distribution, you can still say the probability of y_pred should go up if the loss is small (“success”) or down if it is large (“failure”).</p>

trinity: <p>Thanks for the reply. Sorry, I should have been more precise. By saying ’ it is not possible to do any interpolation in <code>y_true</code>', I meant that it is not possible to generate values of <code>y_true</code> at fractional indices by interpolation, such as <code>y_true[3.5,4.5]</code>. That is why the rounding is needed. So in a nutshell, I was wondering on how to compute the loss with a list (<code>y_pred</code>) of fractional indices. But I am guessing this is getting away from the topic of discussion in this post, I might make a separate post then.</p>