Higher order gradients of CTCLoss

username: mebrunet
cooked: <p>I’m using the CTCLoss in PyTorch version 1.0.0.</p>
<p>My vague understanding from the source and discussions I’ve read is that it wraps some external cpp modules (linking to cudnn), and implements its own <code>backwards()</code> rather than relying on pytorch’s autograd.</p>
<p>I therefore assume higher order gradients (e.g. HVPs) through CTCLoss won’t work.</p>
<p>For example, running:</p>
<pre><code class="lang-auto"># CTC Loss test
import torch

ctc_loss = torch.nn.CTCLoss()
log_probs = torch.randn(50, 16, 20).log_softmax(2).detach().requires_grad_()
log_probs.size()

targets = torch.randint(1, 20, (16, 30), dtype=torch.long)
targets.size()

input_lengths = torch.full((16,), 50, dtype=torch.long)
target_lengths = torch.randint(10,30,(16,), dtype=torch.long)

loss = ctc_loss(log_probs, targets, input_lengths, target_lengths)

(grads, ) = torch.autograd.grad(loss, log_probs, create_graph=True)  # ∇J
grads_summed = torch.sum(grads, (0,1,2))  # equilvant to ∇J⋅1
torch.autograd.grad(grads_summed, log_probs)  # attempt HVP ∇(∇J⋅1)
</code></pre>
<p>results in:</p>
<blockquote>
<p>Runtime Error: derivative for _ctc_loss_backward is not implemented</p>
</blockquote>
<p>Is there any way around this other than to take on the daunting task of implementing <code>backward()</code> for <code>ctc_loss_backward</code>?</p>

username: tom
cooked: <aside class="quote no-group" data-username="mebrunet" data-post="1" data-topic="35019">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mebrunet/48/8800_2.png" class="avatar"> mebrunet:</div>
<blockquote>
<p>I therefore assume higher order gradients (e.g. HVPs) through CTCLoss won’t work.</p>
</blockquote>
</aside>
<p>Indeed. I’d recommend starting from the <a href="http://www.cs.toronto.edu/~graves/icml_2006.pdf" rel="noopener nofollow ugc">original paper</a>, you could differentiate equations 15 and the derivative below it to compute the second derivative. I’ve tried to comment the source to make it easy to follow with the paper in mind (but remember that alpha and beta are computed in log space).<br>
I’m not sure whether it will work well numerically - even for the first derivative of ctc_loss, the numerical precision can be touchy.<br>
I’d be interested to see your results.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ShubhangDesai
cooked: <p><a class="mention" href="/u/mebrunet">@mebrunet</a> Any progress on this? Would love to know if you made any headway on implementing backward for ctc_loss_backward.</p>

username: mebrunet
cooked: <p>I haven’t made the time yet to try… Our work took a turn, and I haven’t needed it since. Will post here if ever I take a crack at it.</p>

username: grazder
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a> ! I’m trying to do double backward for ctc loss. Is there a way to approximate ctc gradient with finite differences? And is there a way to calculate ctc backward in double type so that there are no precision problems</p>

username: tom
cooked: <p>I’m not aware of a double backward for ctc loss being available<br>
By their very nature, finite differences compute approximate directional derivatives (so jacobian-vector products if you want, what you would get with forward mode) rather than gradients (aka vector-jacobian products). As such generally compute-intensive to approximate gradients with finite differences because you would need as many evaluations as you have inputs.<br>
You can use CTC loss with doubles just by passing them, enabling this was one of my goals with having an integrated open source implementation.</p>
<p>Best regards</p>
<p>Thomas</p>