akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to feed different pad IDs to a collate function?

seankala: <p>Hi. I usually use a custom <code>collate_fn</code> and use it as an argument when defining my DataLoader. It usually looks something like:</p>
<pre><code class="lang-auto">def collate_fn(batch):
    max_len = max([len(b['input_ids']) for b in batch])
    input_ids = [b['input_ids'] + ([0] * (max_len - len(b['input_ids'])))]
    labels = [b['label'] for b in batch]
    return input_ids
</code></pre>
<p>As you can see, I’m using <code>0</code> for my padding sequence. What I’m wondering is, since language models and their tokenizers use different IDs for padding tokens, is there a way that I can make the <code>collate_fn</code> flexible to take that into account?</p>

seankala: <p>I was able to make a workaround but making a Trainer class and making the <code>collate_fn</code> a method. After that I was able to do something like <code>self.pad_token_id = tokenizer.pad_token_id</code> and modify the original <code>collate_fn</code> to use <code>self.pad_token_id</code> rather than a hardcoded value.</p>
<p>I’m still curious if there’s any way to do this while keeping <code>collate_fn</code> a top-level function though. For example if there would be any way to pass an argument or something.</p>