Masking input to loss function

huberl: <p>I’m currently implementing pseudo labeling, where I create the labels for the unlabeled part of the datset by simply running the samples trough the model and using the prediction as ground truth. I’m only using the prediction for a sample as ground truth, however, if its confidence surpasses a given <strong>threshold</strong>.</p>
<p>To implement this, I tried using two approaches:</p>
<pre><code class="lang-auto">conf, pseudo_label = F.softmax(out, dim=1).max(axis=1)
mask = conf &gt; threshold

# Option 1
loss = F.cross_entropy(out[mask], pseudo_label[mask])

# Option 2
loss = (F.cross_entropy(out, pseudo_label, reduction='none') * mask).mean()
</code></pre>
<p>Which of them is preferrable? Option 1 produces nan values when the mask is False for every item, but option 2 is way more computationally expensive since it always runs the whole dataset trough the loss function.</p>

huberl: <p>I tried multiple other things:</p>
<pre><code class="lang-auto">tnsr = torch.Tensor([1,2,3,4])
mask = torch.BoolTensor([False, False, True, True])
a = tnsr * mask  # Gives tensor([0., 0., 3., 4.])
b = torch.masked_select(tnsr, mask)   # Gives tensor([3., 4.])
c = tnsr[mask]     # Gives tensor([3., 4.])
</code></pre>
<p>I’m still not sure which of these options to choose.</p>

pascal_notsawo: <p>I would advise you to choose option 1</p>
<aside class="quote no-group" data-username="huberl" data-post="1" data-topic="121830">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huberl/48/38447_2.png" class="avatar"> huberl:</div>
<blockquote>
<pre><code class="lang-auto"># Option 1
loss = F.cross_entropy(out[mask], pseudo_label[mask])
</code></pre>
</blockquote>
</aside>
<p>This is an option I’ve always used to train my language models with the Mask Language Modeling objective: once I mask the tokens in my sentence and pass it to the BERT encoder, I use a mask to select the representations of the masked tokens that I then project to get probabilities.<br>
As you can also see in the original code of XLM (<a href="https://github.com/facebookresearch/XLM" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining.</a>) : <a href="https://github.com/facebookresearch/XLM/blob/master/xlm/model/transformer.py#L438" rel="noopener nofollow ugc">here</a> for the mask selection and <a href="https://github.com/facebookresearch/XLM/blob/master/xlm/model/transformer.py#L134" rel="noopener nofollow ugc">here</a> for the loss computation.</p>

bdzyubak: <p>Thanks for the post. I am new to language models. Can you comment on whether the following is the best way to train?<br>
I am training abstractive question answering with transformers. My labels are company names tokenized to 25 tokens. Output generation is maxed at 25 tokens but sometimes yields less. Can/should I do the following as a rule?</p>
<pre><code class="lang-auto">out = F.pad(out, (1, labels.shape[1] - labels.shape[1] - 1))
mask = out &gt; 0
loss = F.cross_entropy(out[mask], labels[mask])
</code></pre>