F1 weighted score about BERT model in pytorch

username: Yorgos_Pantis
cooked: <p>I have created a function for evaluation a function. It takes as an input the model and validation data loader and return the validation accuracy, validation loss and f1_weighted score.</p>
<pre><code class="lang-auto">def evaluate(model, val_dataloader):
    """
    After the completion of each training epoch, measure the model's performance
    on our validation set.
    """
    # Put the model into the evaluation mode. The dropout layers are disabled during
    # the test time.
    model.eval()

    # Tracking variables
    val_accuracy = []
    val_loss = []
    f1_weighted = []

    # For each batch in our validation set...
    for batch in val_dataloader:
        # Load batch to GPU
        b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

        # Compute logits
        with torch.no_grad():
            logits = model(b_input_ids, b_attn_mask)

        # Compute loss
        loss = loss_fn(logits, b_labels)
        val_loss.append(loss.item())

        # Get the predictions
        preds = torch.argmax(logits, dim=1).flatten()

        # Calculate the accuracy rate
        accuracy = (preds == b_labels).cpu().numpy().mean() * 100
        val_accuracy.append(accuracy)

        # Calculate the f1 weighted score
        f1_metric = F1Score('weighted') 
        f1_weighted = f1_metric(preds, b_labels)

    # Compute the average accuracy and loss over the validation set.
    val_loss = np.mean(val_loss)
    val_accuracy = np.mean(val_accuracy)
    f1_weighted = np.mean(f1_weighted)

    return val_loss, val_accuracy, f1_weighted 
</code></pre>
<p>The function works well without f1 score, but when it’s inside of the function there is the following error</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-48-0e0f6d227c4f&gt; in &lt;module&gt;()
      1 set_seed(42)    # Set seed for reproducibility
      2 bert_classifier, optimizer, scheduler = initialize_model(epochs=4)
----&gt; 3 train(bert_classifier, train_dataloader, val_dataloader, epochs=4, evaluation=True)
      4 
      5 #1. 77.28

2 frames
&lt;__array_function__ internals&gt; in mean(*args, **kwargs)

/usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py in mean(a, axis, dtype, out, keepdims)
   3368             pass
   3369         else:
-&gt; 3370             return mean(axis=axis, dtype=dtype, out=out, **kwargs)
   3371 
   3372     return _methods._mean(a, axis=axis, dtype=dtype,

TypeError: mean() received an invalid combination of arguments - got (out=NoneType, axis=NoneType, dtype=NoneType, ), but expected one of:
 * (*, torch.dtype dtype)
 * (tuple of ints dim, bool keepdim, *, torch.dtype dtype)
 * (tuple of names dim, bool keepdim, *, torch.dtype dtype)
</code></pre>
<p>The core for f1 score can be found here</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/62265351/measuring-f1-score-for-multiclass-classification-natively-in-pytorch">
  <header class="source">

      <a href="https://stackoverflow.com/questions/62265351/measuring-f1-score-for-multiclass-classification-natively-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/7727261/thomas-angeland" target="_blank" rel="noopener nofollow ugc">
    <img alt="Thomas Angeland" src="https://i.stack.imgur.com/5ki3x.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/62265351/measuring-f1-score-for-multiclass-classification-natively-in-pytorch" target="_blank" rel="noopener nofollow ugc">Measuring F1 score for multiclass classification natively in PyTorch</a>
</h4>

<div class="tags">
  <strong>python, validation, machine-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/7727261/thomas-angeland" target="_blank" rel="noopener nofollow ugc">
    Thomas Angeland
  </a>
  on <a href="https://stackoverflow.com/questions/62265351/measuring-f1-score-for-multiclass-classification-natively-in-pytorch" target="_blank" rel="noopener nofollow ugc">03:16PM - 08 Jun 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>