CTC loss comparison

username: Mohammad_Hassan_Soha
cooked: <p>Hi</p>
<p>I’m confused with CTC losses available right now. I think the most popular binding (right now) is <a href="https://github.com/SeanNaren/warp-ctc" rel="nofollow noopener">sean naren warp-ctc binding</a>. But as Sean recommends <a href="https://github.com/pytorch/pytorch/issues/3420" rel="nofollow noopener">here</a>, we should migrate to use <a href="https://github.com/t-vi/warp-ctc" rel="nofollow noopener">this binding</a> instead. Meanwhile native pytorch CTC loss introduced in Pytorch1.0 is another suggestive option.</p>
<p>Is there any fair comparison between these bindings and native Pytorch1.0 CTC? Which one is faster? And what are most notable differences between warp-ctc and Pytorch1.0 CTC implementation? What are pros and cons of using pytorch CTC instead of warp-ctc?</p>
<p><a class="mention" href="/u/seannaren">@SeanNaren</a> <a class="mention" href="/u/tom">@tom</a></p>

username: tom
cooked: <p>PyTorch 1.0 CTC loss is faster.<br>
You’re pretty safe if you use CuDNN loss (you can use <code>print (evaluated_loss.grad_fn)</code> to check if it says something with <em>cudnn</em> or something with <em>native</em>), but there appears to be a bug I haven’t quite hunted down yet when you use the native implementation and inputs of varying length.<br>
I don’t suggest to use my WarpCTC bindings, Sean also has PyTorch 1.0 bindings (I think in a branch, <a class="mention" href="/u/jinserk">@jinserk</a> contributed to them).<br>
Not that PyTorch’s CTC loss isn’t a drop-in replacement for WarpCTC. There is “infinite losses are not clipped / produce NaN gradients” and “PyTorch uses a different scaling when you use the ‘mean’ reduction option”.<br>
I hope to fix the bug mentioned above this year (it’s somewhere in the issue list, but it might be closed)…</p>
<p>Best regards</p>
<p>Thomas</p>

username: SeanNaren
cooked: <p>Agree with what Thomas said!</p>
<p>We’ve ran into the issues Thomas mentioned using the ctc loss in Pytorch 1.0 so still using warp-ctc (which now has 1.0 support). Once these issues have been addressed i’ll migrate deepspeech.pytorch to use the built in loss function.</p>

username: dzubke
cooked: <p>Hello Thomas,</p>
<p>I’m using Awni Hannun <a href="https://github.com/awni/speech" rel="nofollow noopener">codebase</a> to do phoneme recognition which uses his pytorch 0.4 warp-ctc bindings. I have recently started using pytorch’s native ctc loss function in pytorch 1.6, and I noticed that the convergence times for pytorch’s native loss function seem to be about 2-3 times slower than awni’s warp-ctc loss function.</p>
<p>I haven’t done thorough convergence comparisons of the two loss functions, but given your statement that “PyTorch 1.0 CTC loss is faster” I wanted to ask: is the native pytorch ctc loss function is faster than <a href="https://github.com/t-vi/warp-ctc" rel="nofollow noopener">your warp-ctc bindings</a>?</p>
<p>If the pytorch native loss function is faster than your bindings then perhaps there is something in my hyperparameter selection that is making the convergence of the pytorch native loss function slower.</p>
<p>Thanks for your time <a class="mention" href="/u/tom">@tom</a>!</p>