[BUG] ctc_loss makes me confuse

CuriousCat-7: <p>The pred and targes should be on cpu or gpu?<br>
Here is my implementation:</p>
<pre data-code-wrap="python"><code class="lang-python">
def pytorch_ctc_loss(pre, gt, gtlen):
    """
    Input:
        pre   : (T,B,C), FloatTensor
        gt    : (sum(target_lengths)), IntTensor
        gtlen : B, IntTensor
    """
    pre = F.log_softmax(pre, dim=-1)
    prlen = torch.IntTensor([pre.size(0)]*pre.size(1))
    return F.ctc_loss(pre, gt.cpu(), prlen, gtlen.cpu(), blank=0)
</code></pre>
<p>Where “pre” is on cuda.<br>
Mostly, this function works fine, but it will SOMETIMES throw:</p>
<pre><code class="lang-auto">RuntimeError: Tensor for argument #2 'targets' is on CPU, but expected it to be on GPU (while checking arguments for ctc_loss_gpu)
</code></pre>
<p>When the SOMETIMES happend, I pdb it,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/034074df8fbdcb7a9dca80a2a09e33b909316c32.png" data-download-href="https://discuss.pytorch.org/uploads/default/034074df8fbdcb7a9dca80a2a09e33b909316c32" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/034074df8fbdcb7a9dca80a2a09e33b909316c32.png" alt="image" data-base62-sha1="sLwQWP1JJfrM307mI7SLUGlmxA" width="689" height="63" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1223×113 7.96 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and find:</p>
<pre><code class="lang-auto">(Pdb) p log_probs.shape
torch.Size([14, 2, 95])
(Pdb) p targets.shape
torch.Size([20])
(Pdb) p input_lengths
tensor([14, 14], dtype=torch.int32)
(Pdb) p target_lengths
tensor([16,  4], dtype=torch.int32)
(Pdb) p targets
tensor([41, 78, 84, 69, 82, 67, 79, 77, 77, 85, 78, 73, 67, 65, 84, 69, 16, 24,
        18, 17], dtype=torch.int32)
(Pdb) p log_probs
tensor([[[-1.6399, -5.8873, -5.7742,  ..., -5.3770, -6.8403, -5.3823],
         [-2.8857, -5.2404, -5.5605,  ..., -4.9667, -5.4475, -5.8055]],
        [[-2.0398, -5.3816, -4.9946,  ..., -5.6343, -6.3017, -5.2780],
         [-2.8713, -4.9202, -4.8566,  ..., -4.7667, -5.6632, -5.3870]],
        [[-3.1458, -5.6386, -5.0675,  ..., -5.0270, -5.4826, -5.4561],
         [-3.0797, -5.2278, -4.9986,  ..., -4.5412, -5.1524, -5.6291]],
        ...,
        [[-3.0120, -5.4773, -5.3341,  ..., -4.6658, -5.0991, -5.6653],
         [-1.7960, -6.1763, -6.0949,  ..., -4.8088, -6.0141, -5.6578]],
        [[-1.9499, -5.7898, -5.2879,  ..., -4.7644, -5.2784, -5.6835],
         [-0.8562, -6.6550, -5.9762,  ..., -4.9032, -6.6171, -5.8010]],
        [[-1.2654, -6.1024, -5.7864,  ..., -4.8103, -5.6454, -6.5198],
         [-0.7837, -6.3323, -5.9075,  ..., -5.6405, -6.6671, -6.6456]]],
       device='cuda:0', grad_fn=&lt;LogSoftmaxBackward&gt;)
</code></pre>
<p>I can’t see any problem in here.</p>

tom: <p>Yes, that is because it switches between native and cudnn. If you use long tensors (do use <code>torch.tensor(..., dtype=torch.long)</code>, not the constructor…), you’ll always get the native implementation.<br>
We are looking to <a href="https://github.com/pytorch/pytorch/issues/22234" rel="nofollow noopener">improve this situation</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

CuriousCat-7: <p>Thank you for your reply, I have transposed all int tensor to long. I also place log_probs &amp; targets to cuda, place lengths to cpu. But I found that the speed of placing log_probs &amp; targets on cuda or cpu are nearly same. Why?  should I always use cpu?</p>
<pre><code class="lang-python">def pytorch_ctc_loss(pre, gt, gtlen):
    """
    Input:
        pre   : (T,B,C), FloatTensor
        gt    : (sum(target_lengths)), IntTensor
        gtlen : B, IntTensor
    """
    pre = F.log_softmax(pre, dim=-1)
    prlen = torch.LongTensor([pre.size(0)]*pre.size(1))
    return F.ctc_loss(pre, gt.long().to(pre.device), prlen, gtlen.long().cpu(), blank=0)
</code></pre>