Custom loss for huggingface Trainer for sequences

h.l.99: <p>I have a dilemma, for the following custom loss I got this error:<br>
code:</p>
<pre><code>class CustomTrainer(Trainer):
  def compute_loss(self, model, inputs, return_outputs=False):
    use_cuda = torch.cuda.is_available()
    device = torch.device("cuda" if use_cuda else "cpu")
    
    # forward pass
    outputs = model(**inputs)
    logits = outputs.get("logits")
   

    # logits: (batch_size, sequence_length, num_classes), reshape it to (batch_size*sequence_length, num_classes)
    predictions = logits.view(-1, model.config.num_labels)
    # shape: (batch_size*sequence_length)
    predictions = predictions.argmax(dim=-1)

    labels = inputs.get("labels").to(device)
    labels =  labels.view(-1) 

    labels = torch.tensor(labels, dtype=torch.float)
    predictions = torch.tensor(predictions, dtype=torch.float, requires_grad=True)

    loss = loss_fct(predictions, labels)


    return (loss, outputs) if return_outputs else loss
</code></pre>
<p>Error:</p>
<blockquote>
<p>IndexError: Target 17 is out of bounds.</p>
</blockquote>
<p>When I remove these lines of converting dtype:</p>
<pre><code>#labels = torch.tensor(labels, dtype=torch.float)
#predictions = torch.tensor(predictions, dtype=torch.float, requires_grad=True)
</code></pre>
<p>I got this error:</p>
<blockquote>
<p>RuntimeError: Expected floating point type for target with class<br>
probabilities, got Long</p>
</blockquote>

ptrblck: <p>I guess you might be using <code>nn.CrossEntropyLoss</code> as the <code>loss_fct</code>?<br>
If so, note that this criterion accepts model outputs in the shape <code>[batch_size, nb_classes, *]</code> and targets as <code>LongTensor</code>s in the shape <code>[batch_size, *]</code> containing class indices in the range <code>[0, nb_classes-1]</code> as well as <code>FloatTensor</code>s in the same shape as the model output containing probabilities.</p>
<p>Also, these lines looks wrong:</p>
<pre><code class="lang-python">predictions = predictions.argmax(dim=-1)
predictions = torch.tensor(predictions, dtype=torch.float, requires_grad=True)
</code></pre>
<p>since you are detaching the <code>predictions</code> tensor from the computation graph in the <code>argmax</code> operation, which is not differentiable, and it seems you are trying to fix this by re-wrapping the tensor, which will not “re-attach” it to the computation graph.</p>

h.l.99: <p>How to fix that?</p>
<p>I remove this line<br>
<code>predictions = predictions.argmax(dim=-1)</code></p>
<p>and got this error:</p>
<blockquote>
<p>/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)<br>
IndexError: Target 17 is out of bounds.</p>
</blockquote>
<p>the shapes as the following:</p>
<pre><code class="lang-auto">batch_size = 4
seq_length = 128
n_classes = 17
</code></pre>
<p>Before logits shape : <code>torch.Size([4, 128, 17])</code></p>
<p>After predictions shape : <code>torch.Size([512, 17])</code></p>
<p>Before labels shape : <code>torch.Size([4, 128])</code><br>
After labels shape :<code> torch.Size([512])</code></p>
<p>So the input of the CrossEntropyLoss:<br>
predictions shape : <code>torch.Size([512, 17]) : (batch_size * seq_length, n_classes)</code><br>
labels shape : <code>torch.Size([512]): (batch_size * seq_length,)</code></p>

h.l.99: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> as you mentioned the required shape for the <code>CrossEntropyLoss</code></p>
<p>input, I did the following and still get this error:</p>
<blockquote>
<p>IndexError: Target 17 is out of bounds.</p>
</blockquote>
<pre><code class="lang-auto">predictions =logits.view(batch_size, model.config.num_labels, sequence_length)  # shape: (batch_size, n_classes, seq_length)
labels = inputs.get("labels").to(device) #  (batch_size, seq_length)
</code></pre>

InnovArul: <p>Maybe you are already aware of this.<br>
The class labels should be from <code>0</code> to <code>n-1</code>. It seems you are using class labels from 1 to n?</p>

h.l.99: <p>I checked the dictionary <code>model.config.label2id</code> I used for both prepare labels for training, and the classes are from <code>[0-n]</code>: <code>[0-16]</code>.</p>

InnovArul: <aside class="quote no-group" data-username="h.l.99" data-post="4" data-topic="171221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/48/56770_2.png" class="avatar"> h.l.99:</div>
<blockquote>
<p>IndexError: Target 17 is out of bounds.</p>
</blockquote>
</aside>
<p>You may have to check where the indexing value <code>17</code> comes from.</p>

h.l.99: <p>I used the same training data with the default <code>Trainer</code> and it worked without causing this error. I think the default loss is <code>CrossEntropyLoss</code>, so it supposes not to have the target out of bounds.</p>

h.l.99: <p>Also I print the <code>labels</code> inside the <code>compute_loss</code>, and I didn’t see value 17 before the error appear.</p>

h.l.99: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can we do the same functionality of <code>predictions = predictions.argmax(dim=-1)</code> without detaching the <code>predictions</code> tensor from the computation graph?</p>

h.l.99: <p>I have encountered the error of</p>
<blockquote>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>
<p>if I replace <code>argmax</code> with <code>torch.max</code> it shouldn’t detach the tensor from the graph.<br>
<code>_, predictions = torch.max(logits, dim=2)</code>  or even this<br>
<code>_, predictions = torch.max(torch.tensor(logits, requires_grad=True), dim=2)</code></p>
<p>Any idea? <a class="mention" href="/u/ptrblck">@ptrblck</a>  <a class="mention" href="/u/innovarul">@InnovArul</a></p>

ptrblck: <p>No, <code>argmax</code> is not differentiable as already mentioned.<br>
<code>torch.max</code> will return the max values (which are still attached to the computation graph) and the argmax (which will be detached as it’s not differentiable).</p>