Having troubel in understanding what loss is currently in use

username: aquorio15
cooked: <p>I was going through this hugging face code and I am having trouble understanding <strong>what loss the model is currently using</strong> . Although I know most seq2seq models uses CrossEntrophy loss but <strong>I don’t see the definition anywhere in the code</strong></p>
<h4><a name="huggingfacetransformersblobaca6288ff42cebded5421020f0ff088adeb446ddexampleslanguage-modelingrun_clmpyhttpsgithubcomhuggingfacetransformersblobaca6288ff42cebded5421020f0ff088adeb446ddexampleslanguage-modelingrun_clmpy-1" class="anchor" href="#huggingfacetransformersblobaca6288ff42cebded5421020f0ff088adeb446ddexampleslanguage-modelingrun_clmpyhttpsgithubcomhuggingfacetransformersblobaca6288ff42cebded5421020f0ff088adeb446ddexampleslanguage-modelingrun_clmpy-1"></a><a href="https://github.com/huggingface/transformers/blob/aca6288ff42cebded5421020f0ff088adeb446dd/examples/language-modeling/run_clm.py" rel="noopener nofollow ugc">huggingface/transformers/blob/aca6288ff42cebded5421020f0ff088adeb446dd/examples/language-modeling/run_clm.py</a></h4>
<p>Actually I want to train the model with a new custom loss. I have trained a baseline model and its working fine.</p>
<p>Thank You</p>

username: tom
cooked: <p>The transformers library seems to wire the loss into the model’s forward, e.g. Llama it is CrossEntropyLoss on the next token, indeed.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/huggingface/transformers/blob/7293fdc5b9cc809c2aa2ceb84f903ad47e5c06f0/src/transformers/models/llama/modeling_llama.py#L1064-L1069">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/blob/7293fdc5b9cc809c2aa2ceb84f903ad47e5c06f0/src/transformers/models/llama/modeling_llama.py#L1064-L1069" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/huggingface/transformers/blob/7293fdc5b9cc809c2aa2ceb84f903ad47e5c06f0/src/transformers/models/llama/modeling_llama.py#L1064-L1069" target="_blank" rel="noopener nofollow ugc">huggingface/transformers/blob/7293fdc5b9cc809c2aa2ceb84f903ad47e5c06f0/src/transformers/models/llama/modeling_llama.py#L1064-L1069</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1064" style="counter-reset: li-counter 1063 ;">
          <li>loss_fct = CrossEntropyLoss()</li>
          <li>shift_logits = shift_logits.view(-1, self.config.vocab_size)</li>
          <li>shift_labels = shift_labels.view(-1)</li>
          <li># Enable model parallelism</li>
          <li>shift_labels = shift_labels.to(shift_logits.device)</li>
          <li>loss = loss_fct(shift_logits, shift_labels)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am not sure whether I can recommend following that style in modelling.</p>
<p>For easier to understand implementations, A. Karpathy’s <a href="https://github.com/karpathy/nanogpt" rel="noopener nofollow ugc">NanoGPT</a> or Lightning.ai’s <a href="https://github.com/Lightning-AI/lit-gpt" rel="noopener nofollow ugc">LitGPT</a> (disclosure: I do some work for them) might be good choices.</p>
<p>Best regards</p>
<p>Thomas</p>

username: aquorio15
cooked: <p>Thanks for the quick reply, actually currently I am using pre-trained llama models from hugging face. I want to fine-tune llama with a weighted loss function. Any idea how I can integrate it in the transformer library? I found some links related to that but it does not seem to working.</p>

username: tom
cooked: <p>You could subclass the model and reimplement the forward with your modification or so.</p>
<p>However, I do think that fundamentally,  transformers is a library targeted towards people using the models as-is. The other two repos that I linked (and A. Karpathy’s earlier MinGPT both have as a reference point) deliberately want do differently: there, it is intended that you take and modify the code rather than using them just as a library.</p>
<p>Best regards</p>
<p>Thomas</p>