Right vs Left Padding

Basu_Jindal: <p>While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?<br>
Many papers use Left Padding, but is right padding wrong since transformers gives the following warning if using right padding " A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set <code>padding_side='left'</code> when initializing the tokenizer."</p>

vdw: <p>I would simply assume the model was pretrained with left-padded input, so it seems reasonable to adhere to it for fine-tuning.</p>
<p>Left-padding feels also more intuitive <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Basu_Jindal: <p>Thanks for the reply, foundational models like LLaMA are usually trained without padding.<br>
It is mostly used during finetuning, also can you please tell why left padding feels more intuitive? Thanks!</p>

2802701695yyb: <p>I’m concerned about this as well, and the generate() method in the transformers library explicitly suggests that decoder-only models should use the left padding method. I would also like to know the reason for this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d99996bb8890d92976ba764e911ef804a4db707.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d99996bb8890d92976ba764e911ef804a4db707" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d99996bb8890d92976ba764e911ef804a4db707_2_690x368.png" alt="image" data-base62-sha1="dm1umxmtRZJcVSulkDfMF9xhwN1" width="690" height="368" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d99996bb8890d92976ba764e911ef804a4db707_2_690x368.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d99996bb8890d92976ba764e911ef804a4db707_2_1035x552.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d99996bb8890d92976ba764e911ef804a4db707.png 2x" data-dominant-color="30333A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1234×659 107 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Basu_Jindal: <p>I asked the same question on Stack overflow and got a good answer.<br>
<a href="https://ai.stackexchange.com/questions/41485/while-fine-tuning-a-decoder-only-llm-like-llama-on-chat-dataset-what-kind-of-pa" rel="noopener nofollow ugc">transformer - While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use? - Artificial Intelligence Stack Exchange</a></p>

ibrahimciko: <p>I have been thinking about this for quite some time. How do they manage to train without padding if they train with batch of variable length observations?. Or maybe, they just chunk the whole text corpus into size of equal lengths and train like that. In that setup, there is no longer the problem of variable input length. However, is this a correct approach? What if the semantically related sentence ended up in the other batch?</p>

Basu_Jindal: <p>It is the latter, for pre-training LLMs they collect a large corpus and as you said, divide it into segments of equal length. This can bundle two unrelated paragraphs together but when because the models are trained on trillions of tokens it doesn’t affect the model much. Also, there is the other option of masking out the unrelated text.</p>

Mauro_Escudero: <p>It’s my understanding that during pre-training, the corpus is “packed” into examples of the same length and for that reason non-instruct models like Llama-3.2-1B do not need a pad token. You can verify this by checking that <code>tokenizer.pad_token</code> is <code>None</code>.</p>
<p>The key question is, what do we do when we want to instruction fine-tune this model? Several threads/conversations I have read online suggest doing this:</p>
<pre data-code-wrap="python"><code class="lang-python">tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-3.2-1B",
    chat_template=chat_template,
    padding_side="left"
)
if tokenizer.pad_token is None:
    tokenizer.pad_token = "&lt;|finetune_right_pad_id|&gt;"
    tokenizer.add_special_tokens({'pad_token': "&lt;|finetune_right_pad_id|&gt;"})
</code></pre>
<p>that is, setting the pad token as one of several already available special tokens. In particular, rather than using the reserved ones, we can use the <code>"&lt;|finetune_right_pad_id|&gt;"</code>. They also suggest padding on the left, meaning that the text will look as follows:</p>
<pre><code class="lang-auto">&lt;bos&gt;&lt;pad&gt;&lt;pad&gt;{{text}}
</code></pre>
<p>It’s unclear to me why this is needed. Isn’t it the case that if EOS tokens were on the right, e.g.</p>
<pre><code class="lang-auto">&lt;bos&gt;{{text}}&lt;eos&gt;&lt;eos&gt;
</code></pre>
<p>then the model would just stop at the first EOS?</p>