Loss not improving when training t5 transformer on SORIE for Question Answering

bdzyubak: <p>Dear community,</p>
<p>I am trying to fine tune T5 for question answering on the SROIE (textlines of receipts) dataset but the loss is not improving. Can you offer a suggestion? My text is OCR-extracted textlines from receipts which have bounding box coordinates and free text. My questions are to extract the company where the purchase was made and the total amount spent. The label (e.g. Total) and value ($10) are not necessarily in the same text line/not necessarily adjacent in the concatenated text. I believe, the model should be able to use the bounding box coordinates to interpret relationships between textlines. Is this a correct assumption?</p>
<p>My prompt is like this:</p>
<blockquote>
<p>‘Context: 68,266,689,266,689,303,68,303,PANDAH INDAH PULAU KETAM RESTAURANT\n104,310,667,310,667,347,104,347,M4-A-8, JALAN PANDAH INDAH … \n\n\nQuestion: What is the total amount spent?\n\nAnswer:’</p>
</blockquote>
<pre><code class="lang-auto">tokens_prompt_total = self.tokenizer(prompt_total,
                                             padding="max_length",
                                             max_length=3000)
</code></pre>
<p>My label is like this:</p>
<blockquote>
<p>‘PANDAH INDAH PULAU KETAM RESTAURANT’</p>
</blockquote>
<pre><code class="lang-auto">tokens_labels_company = self.tokenizer(labels_dict['company'],                                        padding="max_length",
max_length=25).input_ids
</code></pre>
<p>My loss is calculated like this, but the value of it is very high and doesn’t decrease.</p>
<pre><code class="lang-auto">        label_tokens_for_loss_calc = torch.stack(labels_tokens, dim=1).type(torch.FloatTensor).to('cuda')
        output_tokens_for_loss_calc = outputs.type(torch.FloatTensor).to('cuda')
        output_tokens_for_loss_calc = F.pad(output_tokens_for_loss_calc, (1,
                                                                          label_tokens_for_loss_calc.shape[1] -
                                                                          output_tokens_for_loss_calc.shape[1] - 1))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c943683436178c6db56169484a923b459458df8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/c943683436178c6db56169484a923b459458df8a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c943683436178c6db56169484a923b459458df8a_2_690x402.png" alt="image" data-base62-sha1="sIsod36ShCwmT5pjdEGJKlGDHtg" width="690" height="402" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c943683436178c6db56169484a923b459458df8a_2_690x402.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c943683436178c6db56169484a923b459458df8a_2_1035x603.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c943683436178c6db56169484a923b459458df8a_2_1380x804.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2419×1411 181 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So far, I am trying to fine tune the whole model without freezing since bounding box comprehension is somewhat unique to this task, but could try freezing or using another model. Greatly appreciate your suggestions!</p>