Differentiable slicing operation with float index

sndjn: <p>I have a 1D tensor with audio data, now I would like to train my neural network to create an index in the form of a float with which I can slice a window from the beginning of this index + 512 from the audio data to calculate a loss. Unfortunately, I have not yet found a way to do this without having to convert the float index into an int, which prevents a gradient calculation.</p>

KFrank: <p>Hi sndjn!</p>
<aside class="quote no-group" data-username="sndjn" data-post="1" data-topic="199886" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/3da27b/48.png" class="avatar"> sndjn:</div>
<blockquote>
<p>create an index in the form of a float with which I can slice a window</p>
</blockquote>
</aside>
<p>You probably won’t be able to make this work.  The basic problem is that<br>
such an index is discrete, and, as such, is not (usefully) differentiable.</p>
<aside class="quote no-group">
<blockquote>
<p>I have not yet found a way to do this without having to convert the float index into an int, which prevents a gradient calculation.</p>
</blockquote>
</aside>
<p>Pytorch knows that the operation of converting a float into an int is not<br>
(usefully) differentiable and does not permit <code>requires_grad = True</code> to<br>
be attached to an int (or a <code>LongTensor</code>, which it th specific type of tensor<br>
you would use as an index).</p>
<p>Here’s the problem:  Let’s say that your float index is <code>7.41</code>.  Perhaps you<br>
round it to <code>7</code> to get an int index.  As your float varies from <code>7.0</code> to <code>7.5</code>,<br>
the rounded value, <code>7</code>, remains constant, so the derivative is zero.  Then<br>
at <code>7.5</code> the rounded value pops up to <code>8</code>.  (Right at this point, the rounding<br>
function is technically not differentiable.)  Then from <code>7.5</code> to <code>8.0</code>, the<br>
rounded value, <code>8</code>, is constant again, and the derivative is again zero.</p>
<p>While this function is differentiable “almost everywhere”, with derivative<br>
zero, a derivative of zero isn’t useful for training with a gradient-descent<br>
optimizer, because it doesn’t tell the optimizer in what direction is should<br>
move the model parameters.  This is what I mean by “not usefully<br>
differentiable” (and why pytorch doesn’t support <code>requires_grad</code> for<br>
integer tensors).</p>
<p>However (and this might not make sense or work for your use case),<br>
you could possibly use a float “index” to interpolate into your audio data.</p>
<p>The idea would be that for an index of <code>7.0</code>, you would use your audio<br>
values given by <code>data[7], data[8], data[9], ...</code>.  But for an “index”<br>
of <code>7.41</code>, you would use <code>di (7.41), di (8.41), di (9.41), ...</code>,<br>
where, for example, <code>di (7.41)</code> is the value obtained by interpolating<br>
between <code>data[7]</code> and <code>data[8]</code>.</p>
<p>Provided that your interpolation scheme (for example, linear interpolation)<br>
is (usefully) differentiable (and the subsequent processing that turns your<br>
interpolated data into a loss is also differentiable), you will be able to<br>
backpropagate through your float “index” and train your model.</p>
<p>(This isn’t the only possible scheme:  You could, for example, just use<br>
interpolation for the first and last data points in the window.  Or you<br>
could interpolate between (use a weighted average of) the loss value<br>
you get from starting your window at <code>7</code> and the one you get from starting<br>
your window at <code>8</code>.  And so on …)</p>
<p>Good luck!</p>
<p>K. Frank</p>

sndjn: <p>Thank you for your answer, I have found a solution that will probably work for my application. I have programmed a simplified example with only one float weight for optimization:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1F9pNDkx0j5xt_tkfNgggttbkWJDEojsE?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/a3becc95c1f5fd483ef8052ea2b002ed/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1F9pNDkx0j5xt_tkfNgggttbkWJDEojsE?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1F9pNDkx0j5xt_tkfNgggttbkWJDEojsE?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>For those who only want the solution:</p>
<pre data-code-wrap="python"><code class="lang-python">@torch.jit.script
def torch_interp(indices, xp, fp):
    pos = torch.searchsorted(xp, indices, right=True)
    pos = pos.clamp(min=1, max=xp.size(0)-1)
    left = xp[pos - 1]
    right = xp[pos]
    weight = (indices - left) / (right - left)

    return (1 - weight) * fp[pos - 1] + weight * fp[pos]
test = F.pad(signal, (0, 512))
float_weight = torch.tensor([(torch.rand(())-0.5)*20.0], requires_grad=True) # initialized with random float
target_float = 0.1 # to slice from the test tensor for loss calculation and to compare with the optimized float_index
window_size = 512   # window_size for comparison
optimizer = torch.optim.AdamW([float_weight], lr=0.1)
start_i = int(target_float*target_length)

loss_fn = SoftDTWLossPyTorch(gamma=0.1,normalize=True)

for epoch in range(1000):
    optimizer.zero_grad()
    
    float_index = torch.sigmoid(float_weight)

    indices = torch.arange(0, window_size) + (float_index*target_length)
    xp = torch.arange(test.size(0), dtype=torch.float32)
    x = torch_interp(indices, xp, test)
    
    # Loss-Calculation
    loss = loss_fn(x.unsqueeze(0).unsqueeze(-1),test[start_i:start_i+window_size].unsqueeze(0).unsqueeze(-1)) 
    # Much faster loss function that also works, but not as well: (x-test[start_i:start_i+window_size]).abs().sum()

                # regularization term               # prevent local optima with randomness: probably unnecessary or even harmful for real networks with different inputs
    loss = loss + loss * (float_weight-0.5).abs() + loss * torch.rand(())*6
    
    loss.backward()
    optimizer.step()
    
    if epoch % 100 == 0:
        print(f'Epoch {epoch}, Loss: {loss.item()}, Float Index: {float_index.item()}')
    if (target_float-float_index).abs()&lt;0.001:
        print("Done")
        break
</code></pre>