Inf and NaN issue when training CRNN

username: Aldemaro14
cooked: <p>Hello, I’m using the CRNN architecture from <a href="https://github.com/carnotaur/crnn-tutorial" class="inline-onebox" rel="noopener nofollow ugc">GitHub - carnotaur/crnn-tutorial: Training Convolutional Recurrent Neural Network (CRNN) using PyTorch</a> for a OCR app, I’m also using custom data, but when the training begins, I got this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73ff235d41167c0b2f0929223377eee661ccecb1.png" data-download-href="https://discuss.pytorch.org/uploads/default/73ff235d41167c0b2f0929223377eee661ccecb1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73ff235d41167c0b2f0929223377eee661ccecb1.png" alt="image" data-base62-sha1="gy9xtWQjyihwYfoRnTtznz4NFip" width="690" height="197" data-dominant-color="E6E6E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">750×215 13.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here some snippets from my notebook:</p>
<pre><code class="lang-auto">class CaptchaDataset(Dataset):
   
    def __init__(self, img_dir: str):
        pathes = os.listdir(img_dir)
        abspath = os.path.abspath(img_dir)
        self.img_dir = img_dir
        self.pathes = [os.path.join(abspath, path) for path in pathes]
        self.list_transforms = transforms.Compose([transforms.Resize((50, 200)), transforms.ToTensor()])
        
    def __len__(self):
        return len(self.pathes)
    
    def __getitem__(self, idx):
        labels = '../datasets/labels'
        path = self.pathes[idx]
        name = self.get_file_name(path)
        full_path= os.path.dirname(os.path.abspath(path))
        
        label_path = labels + '/' + name + '.txt'
        
        text = self.get_current_index_label(label_path)
        img = Image.open(path).convert('RGB')
        img = self.transform(img)
        return img, text
    
    def get_current_index_label(self, path):
        ext = os.path.basename(path).split('.')[1].lower().strip()
        
        f = open(path, 'r')
        contents = f.read()
        return str(contents)
    
    def transform(self, img) -&gt; torch.Tensor:
        return self.list_transforms(img)
    
    def get_file_name(self, path: str) -&gt; str:
        return os.path.basename(path).split('.')[0].lower().strip()


alphabet = """abcdefghijklmnñopqrstuvwxyz0123456789()_,.:*/'-ÃD#±+@ """
label_converter = strLabelConverter(alphabet)
alphabet

hidden_size = 256
vocab_size = len(alphabet) # extra character for blank symbol
bidirectional = True
dropout = 0.1
weight_decay = 1e-5
momentum = 0.9
clip_norm = 5
max_epoch = 50
print(vocab_size)

imgs, texts = iter(val_dataloader).next()
print(imgs.shape, len(texts))
torch.Size([8, 3, 50, 200]) 8

def weights_init(m):
    classname = m.__class__.__name__
    if type(m) in [nn.Linear, nn.Conv2d, nn.Conv1d]:
        torch.nn.init.xavier_uniform_(m.weight)
        m.bias.data.fill_(0.01)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

class CRNN(nn.Module):
    def __init__(self, hidden_size: int, 
                 vocab_size: int, 
                 bidirectional: bool = True, 
                 dropout: float = 0.5):
        super(CRNN, self).__init__()
        self.hidden_size = hidden_size
        self.vocab_size = vocab_size
        self.bidirectional = bidirectional
        # make layers
        # convolutions 
        resnet = resnet18(pretrained=True)
        modules = list(resnet.children())[:-3]
        self.resnet = nn.Sequential(*modules)

        self.cn6 = blockCNN(256, 256, kernel_size=3, padding=1)
        # RNN + Linear
        self.linear1 = nn.Linear(1024, 256)
        self.gru1 = blockRNN(256, hidden_size, hidden_size,
                             dropout=dropout, 
                             bidirectional=bidirectional)
        self.gru2 = blockRNN(hidden_size, hidden_size, vocab_size,
                             dropout=dropout,
                             bidirectional=bidirectional)
        self.linear2 = nn.Linear(hidden_size * 2, vocab_size)
        
    def forward(self, batch: torch.Tensor):
        """
        ------:size sequence:------
        torch.Size([batch_size, 3, 50, 200]) -- IN:
        torch.Size([batch_size, 256, 4, 13]) -- CNN blocks ended
        torch.Size([batch_size, 13, 256, 4]) -- permuted 
        torch.Size([batch_size, 13, 1024]) -- Linear #1
        torch.Size([batch_size, 13, 256]) -- IN GRU 
        torch.Size([batch_size, 13, 256]) -- OUT GRU 
        torch.Size([batch_size, 13, vocab_size]) -- Linear #2
        torch.Size([13, batch_size, vocab_size]) -- :OUT
        """
        batch_size = batch.size(0)
        # convolutions
        batch = self.resnet(batch)
        batch = self.cn6(batch, use_relu=True, use_bn=True)
        # make sequences of image features
        batch = batch.permute(0, 3, 1, 2)
        n_channels = batch.size(1)
        batch = batch.view(batch_size, n_channels, -1)
        batch = self.linear1(batch)
        # rnn layers
        batch = self.gru1(batch, add_output=True)
        batch = self.gru2(batch)
        # output
        batch = self.linear2(batch)
        batch = batch.permute(1, 0, 2)
        return batch

crnn = CRNN(hidden_size=hidden_size, vocab_size=vocab_size, 
            bidirectional=bidirectional, dropout=dropout).to(device)
crnn(imgs.to(device)).shape

lr = 0.01
optimizer = torch.optim.SGD(crnn.parameters(), lr=lr, nesterov=True, 
                            weight_decay=weight_decay, momentum=momentum)
critertion = nn.CTCLoss(blank=0)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, verbose=True, patience=5)

def validation(model, val_losses, label_converter):
    with torch.no_grad():
        model.eval()
        for batch_img, batch_text in val_dataloader:
            logits = crnn(batch_img.to(device))
            val_loss = calculate_loss(logits, batch_text, label_converter)
            val_losses.append(val_loss.item())
    return val_losses

def calculate_loss(logits, texts, label_converter):
    # get infomation from prediction
    device = logits.device
    input_len, batch_size, vocab_size = logits.size()
    # encode inputs
    logits = logits.log_softmax(2)
    encoded_texts, text_lens = label_converter.encode(texts)
    logits_lens = torch.full(size=(batch_size,), fill_value=input_len, dtype=torch.int32)
    # calculate ctc
    loss = critertion(logits, encoded_texts, 
                      logits_lens.to(device), text_lens)
    return loss

train_losses = []
val_losses = []
val_epoch_len = len(val_dataset) // BATCH_SIZE
val_epoch_len

try:
    while epoch &lt;= max_epoch:
        crnn.train()
        for idx, (batch_imgs, batch_text) in enumerate(train_dataloader):
            optimizer.zero_grad()
            logits = crnn(batch_imgs.to(device))
            # calculate loss
            train_loss = calculate_loss(logits, batch_text, label_converter)
            if np.isnan(train_loss.detach().cpu().numpy()):
                continue
            train_losses.append(train_loss.item())
            # make backward
            train_loss.backward()

            nn.utils.clip_grad_norm_(crnn.parameters(), clip_norm)
            optimizer.step()

        val_losses = validation(crnn, val_losses, label_converter)
        
        # printing progress
        plot_loss(epoch, train_losses, val_losses)
        print_prediction(crnn, val_dataset, device, label_converter)
        
        scheduler.step(val_losses[-1])
        epoch += 1
except KeyboardInterrupt:
    pass`Preformatted text` ```

please help.</code></pre>

username: ptrblck
cooked: <p>I would recommend to add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of your script, which would print a stack trace showing pointing towards the operation, which created the first NaN output.<br>
This should be helpful in debugging the issue.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging a bit easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Aldemaro14
cooked: <p>Thank you very much for your support, I’m trying to figure the gradient pout as I’m kinda new to Pytorch, also did what you recommended for the code above.</p>

username: Aldemaro14
cooked: <p>While performing the debugging fount this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18ed86687dd6ba48a66a73325511ba33d6668c0b.png" data-download-href="https://discuss.pytorch.org/uploads/default/18ed86687dd6ba48a66a73325511ba33d6668c0b" title="logits"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18ed86687dd6ba48a66a73325511ba33d6668c0b.png" alt="logits" data-base62-sha1="3ywlY1uafpqPNE0rBrl60iIHFhx" width="629" height="500" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">logits</span><span class="informations">835×663 7.74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Aldemaro14
cooked: <p>Hello once again, after some debugging, I’d notice that my tensors become nan on CTCloss, for some reason unknown to me, I already try tweaking the CTCloss params, and got some good inputs, but they become nan within the first batch.</p>
<p>please help.</p>

username: Aldemaro14
cooked: <p>I was able to fix the problem by adding +1 to the alphabet variable, thanks!!!</p>