Modified CE Loss function

username: Hadjer13
cooked: <p>I’m using a code from github for multi-class classification that uses a modified cross entropy loss function, that discards one class from the loss (first class).<br>
Could someone please recognize the type of CE loss used ? and give me its name if it has one, or a resource to read about it ?</p>
<p>I could know that <code>per_example_loss</code> is a CE loss without the class ‘0’<br>
Could someone tell me what ‘rc_loss’ refers to ?</p>
<pre><code class="lang-auto">probabilities = F.softmax(logits, dim=-1)
                log_probs = F.log_softmax(logits, dim=-1)
                
                print("labels",labels)
                one_hot_labels = F.one_hot(labels, num_classes=self.num_labels)
                if device &gt;= 0:
                    one_hot_labels = one_hot_labels.to(device)
                
                print("hot labels", one_hot_labels)
                dist = one_hot_labels[:, 1:].float() * log_probs[:, 1:]
                print("hot labels except others", one_hot_labels[:, 1:].float())
                print("log prob except others", log_probs[:, 1:])

                example_loss_except_other, _ = dist.min(dim=-1)
                
                print("example loss except others", example_loss_except_other)
                per_example_loss = - example_loss_except_other.mean()

                rc_probabilities = probabilities - probabilities * one_hot_labels.float()
                print("rc_prob ",rc_probabilities )
                second_pre,  _ = rc_probabilities[:, 1:].max(dim=-1)
                print("seconde prob ",second_pre)
                rc_loss = - (1 - second_pre).log().mean()
                print("rc loss ",rc_loss)

                #print(loss, per_example_loss, rc_loss)
                loss += per_example_loss + 5 * rc_loss
</code></pre>
<p>Thanks in advance !</p>

username: harsha_g
cooked: <p>My impression of the 0th class you are referring to is the [CLS] token appended at the beginning of the sequence before feeding it to BERT. Also, I presume RC in rc_probabilities merely stands for Relation Classification. Did you come across this work<br>
<a href="https://arxiv.org/pdf/1905.08284.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1905.08284.pdf</a></p>
<p>while working on the code? This may have more information on what you are looking for.</p>

username: Hadjer13
cooked: <p>I am actually using the <a href="https://github.com/wang-h/bert-relation-classification" rel="nofollow noopener">implementation</a> of the paper you mentioned,<br>
From my understanding, <code>per example loss</code> gives the cross entropy loss without the class ‘Others’ of the dataset (I tried to print <code>one_hot_labels[:, 1:]</code> to make use that it exclude the class Others)<br>
What I could’nt understand is, why did he add the rc_probabilities/rc_probabilities ? how will it affect the loss function ?<br>
Is it a specific variant of the cross entropy loss ?<br>
Thanks in advance</p>

username: harsha_g
cooked: <p>Okay, this took a while to figure out (although not completely). The problem being: the paper says one thing but the code does something else <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:">. Take a look at the other paper by the same authors (at the very bottom of the README.md in the link you posted) and could probably explain why they coded the loss function slightly different from the paper (in fact they call it the rank loss function). I hope this helps.</p>

username: Hadjer13
cooked: <p>YES this helps a lot thank you !<br>
Indeed, it’s a ranking loss</p>