CTCLoss predicts blanks

PersiQ: <p>Hi,</p>
<p>I am doing seq2seq where the input is a sequence of images and the output is a text (sequence of token words). My model is a pretrained CNN layer + Self-attention encoder (or LSTM) + Linear layer and apply the logSoftmax to get the log probs of the classes + blank label (batch, Seq, classes+1) + CTC.<br>
I am using the ctc_loss of Pytorch. i am padding all sequences with the blank token = 0. I followed all the instructions in here (<a href="https://pytorch.org/docs/stable/nn.html#ctcloss" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#ctcloss</a>).<br>
When training, my model seems to predict only blanks after a few batches. And the loss is slowly decreasing and stays very high even after many epochs. After many many epochs, the model produces some non-blank tokens (i am doing torch.max(output, dim=-1) to get the predictions).<br>
When using an encoder-decoder approach completely removing the CTC layer, i get a decent bleu score but horrible WER score (~80).</p>
<p>y shape = torch.Size([2, 13])<br>
y = tensor([[  1,  55,   9, 413, 344,  29, 318,  38,  15, 305, 196, 144,  54],<br>
[  1, 217, 163,   4, 222,  93,  45,  54,   0,   0,   0,   0,   0]],<br>
device=‘cuda:0’)</p>
<p>(Seq, batch, classes+blank)<br>
output = torch.Size([54, 2, 1232])</p>
<p>x_lengths = tensor([44, 54], dtype=torch.int32)<br>
y_lengths = tensor([13,  8], dtype=torch.int32)</p>
<p>loss = ctc_loss(output, y, x_lengths.cpu(), y_lengths.cpu())</p>
<p>I am not sure if this is normal as i heard that it is pretty hard to train with a CTC loss or else i am doing something wrong.</p>
<p>I appreciate you taking the time to read this. I appreciate any feedback on this.<br>
Thank you.</p>

IliasPap: <p>You should not pad target sequences with blank, that’s the main reason your model predicts blanks.<br>
In general is harder to train end-2-end the full architecture only with CTC Loss depending on the model architecture and depth<br>
Also try to use reduction=‘mean’ and zero_infinity=“True” , I found it works better with these parameters.</p>

PersiQ: <aside class="quote no-group" data-username="IliasPap" data-post="2" data-topic="66928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iliaspap/48/14391_2.png" class="avatar"> IliasPap:</div>
<blockquote>
<p>use reduction=‘mean’ and zero_infinity=“True”</p>
</blockquote>
</aside>
<p>hey sorry for the late response and thank you for your reply.</p>
<p>I tried using use reduction=‘mean’ and zero_infinity=“True” but they didn’t work.<br>
It seems that by doing this : loss = ctc_loss(output, y.cpu(), x_lengths.cpu(), y_lengths.cpu()) and updating Pytorch, it made it work perfectly now.</p>

IliasPap: <p>CTC loss calculation can also be done in cuda device, there is no need to send tensors to cpu</p>

Yan_Chu: <p>Hi, I meet the same error.  I pad target labels with “PAD_ID=1”, and “BLANK_ID = 0” in my vocabulary.  After some batches, it only predicts blank.</p>
<pre><code class="lang-auto">def CTCLoss(ctc_decoder_out, samples):
    ctcprobs = ctc_decoder_out.log_softmax(-1)
    loss_func = torch.nn.CTCLoss(blank=0, reduction="mean", zero_infinity=False)
    ctc_target = samples["real_target"]
    src_lengths = samples["src_lengths"]
    tgt_lengths = ctc_target.ne(tgt_dict.pad()).sum(-1)
    ctc_loss = loss_func(ctcprobs, ctc_target.cpu(), src_lengths.cpu(), tgt_lengths.cpu())
    return ctc_loss
</code></pre>
<p>This is my code.  I don’t know weather it is hard to train ctc loss or there are something wrong in my code?  After one epoch, it stills only predict blank and the ctc loss is really slowly decreased. Could you help solve my problem by any chance? Thanks.</p>

PersiQ: <p>Hi Yan,</p>
<p>Make sure your src lengths are long enough in which src lengths &gt;= target lengths or simply try using zero_infinity=True to avoid having infinite losses.<br>
Other than that, training with CTC is a bit tricky, so expect to get a lot of blanks at early epochs as that is perfectly normal.</p>
<p>Hope that helps.</p>

Yan_Chu: <p>Hi PersiQ,</p>
<p>Thank you for your replay. I am sure my src lengths are longer than target lengths.  I tried using both reduction=“sum” and reduction=“mean”, but they both didn’t work.<br>
My task is video based translation, maybe it is difficult to train with ctc loss.  My model is pretrained EfficientNet + self-attention + Linear layer, and then using CTCLoss. During training, after 1 epoch the ctc loss is approximately 5.5(the loss printed is divided by the tokens), and after 30 epochs, the ctc loss is still about 5.2. In addition, the model still only predicts blank.<br>
Do you have experience in training similar models? I hope you can provide some suggestions.</p>
<p>Thank you.</p>

Ahmad_Mughrabi: <p>Hi All,</p>
<p>I tried the proposed solutions on this thread, but , Unfortunately, I still get blank predictions, any idea?</p>
<p>P.S. I see changing the learning rate could change the appearance of the blank predictions.</p>
<p>The code is here: <a href="https://github.com/cwig/start_follow_read/blob/e9a4f7cb92a63e803c958239be4b907f35efa346/hw_pretraining.py#L65" rel="noopener nofollow ugc">https://github.com/cwig/start_follow_read/blob/e9a4f7cb92a63e803c958239be4b907f35efa346/hw_pretraining.py#L65</a></p>
<p>Thanks a million,<br>
Mughrabi</p>

Antoine_Chauviere: <p>Hi all,</p>
<p>I don’t understand how to pad the targets in a batch.</p>
<p>If i have blank=0 and vocabulary=1:N so i have num_class in my model equal to N, so what is my padded value?</p>
<p>I hope i’m clear, thanks<br>
Antoine</p>

ID56: <p>Your padded value is your blank. You can simply do:</p>
<pre><code class="lang-python">import torch

labels = [
    torch.Tensor([1, 2]), 
    torch.Tensor([3, 1, 2]),
    torch.Tensor([2])
]

padded_label_batch = torch.nn.utils.rnn.pad_sequence(
    sequences=labels,
    batch_first=True,
    padding_value=0,
).long()

&gt;&gt;&gt; print(padded_label_batch)
tensor([[1, 2, 0],
        [3, 1, 2],
        [2, 0, 0]])
</code></pre>
<p>That is, your list (batch) of variable sized labels will be padded at the trailing end to the length of the longest label in the batch. A good place to do this operation is inside the collate function of the dataloader.</p>
<p>Later, for CTC loss, you will need the length of the labels. That can be computed with <code>count_nonzero</code> easily, since the padding was zero.</p>
<pre><code class="lang-python">label_lengths = torch.count_nonzero(padded_label_batch, axis=1)
&gt;&gt;&gt; print(label_lengths)
tensor([2, 3, 1])
</code></pre>

Antoine_Chauviere: <p>Ok thank you Mashrur Morshed, it’s very clear.</p>
<p>However, I can’t get my model to converge.</p>
<p>I have 1000 training data and 100 in validation.<br>
Maybe my network is not deep enough, or I should use the pre-trained weights from imagenet, or maybe I just don’t have enough data… If you have any advice, I’ll take it .</p>
<p>I have about 10/100 perfect match after 100 epochs and an LR=0.0001</p>
<p>My model is the following (sorry for some comments in French :p) :</p>
<p>class CRNN(nn.Module):<br>
def <strong>init</strong>(self, num_classes):<br>
super(CRNN, self).<strong>init</strong>()</p>
<pre><code>    self.conv1 = nn.Conv2d(3, 32, kernel_size=(3, 3))
    self.norm1 = nn.InstanceNorm2d(32)
    self.conv2 = nn.Conv2d(32, 32, kernel_size=(3, 3), stride=2)
    self.norm2 = nn.InstanceNorm2d(32)
    self.conv3 = nn.Conv2d(32, 64, kernel_size=(3, 3))
    self.norm3 = nn.InstanceNorm2d(64)
    self.conv4 = nn.Conv2d(64, 64, kernel_size=(3, 3), stride=2)
    self.norm4 = nn.InstanceNorm2d(64)
    
    self.linear1 = nn.Linear(320, 128)
    self.gru = nn.GRU(128, 32, 2, batch_first=True, bidirectional=True)
    self.fc = nn.Linear(64, num_classes)
    
def forward(self, x):
    batch_size = len(x)
    
    ## Convolution Part 
    out = self.conv1(x)
    out = self.norm1(out)
    out = nn.functional.leaky_relu(out)
    
    out = self.conv2(out)
    out = self.norm2(out)
    out = nn.functional.leaky_relu(out)
    
    out = self.conv3(out)
    out = self.norm3(out)
    out = nn.functional.leaky_relu(out)
    
    out = self.conv4(out)
    out = self.norm4(out)
    out = nn.functional.leaky_relu(out) #1, 64, 5, 29
    
    ## Reshape for GRU
    out = out.permute(0, 3, 1, 2) #1, 29, 64, 5 (on met channels et width en first)
    out = out.view(batch_size, out.size(1), -1) #1, 29, 320 (pour chaque width on a 320 values)
    
    ## On branche une linear pour diminuer la taille
    out = self.linear1(out) #1, 29, 128 
    
    ## Apply GRU
    out, _ = self.gru(out) #1, 29, 64 (double because bidirectionnal)
    out = torch.stack([nn.functional.log_softmax(self.fc(out[i]), dim=-1) for i in range(out.shape[0])])
    return out</code></pre>

ID56: <p>How is your model’s performance on the training data? If training loss isn’t converging, then your model could be underfitting. I feel like you could try increasing the layers in the GRU (e.g. to 4), if that is the case. (Then later if it is overfitting, you could regularize by turning on the dropout argument in GRU, i.e. adding dropout between GRU layers).</p>
<p>1000 data for training is also quite low, especially if you’re training from scratch and not finetuning; there’s not enough data to generalize on validation/test. If getting more data isn’t possible, perhaps a pretrained CNN extractor might help (e.g. some model from <a href="https://github.com/rwightman/pytorch-image-models" rel="noopener nofollow ugc">timm</a>. Timm also has a <a href="https://rwightman.github.io/pytorch-image-models/feature_extraction/" rel="noopener nofollow ugc">guide on feature extraction</a> from their models which you might find helpful).</p>
<p>Also are you applying any augmentation methods on your training data?</p>

Antoine_Chauviere: <p>Thanks, after 100 epoch my loss function is around 0.2 (starting with a value of 9).</p>
<p>When I run my model on the training data, I only have 22 perfect predictions out of the 1000…</p>
<p>The errors are quite close to reality, for example:<br>
true: elecsprint<br>
pred: eleecsprint</p>
<p>Yes I have to try a pre-trained model to extract the features and no, I am not using data augmentation at the moment.</p>
<p>It’s not easy to get more data, I have to annotate manually.</p>