Multiclass evaluation not working
Author: puifais
Pull request: no

Hello,

I am new to Transformers library and I'm trying to do Sequence Classification. I have 24 labels and I am getting the following error:

> ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

Even though I already added the averaging method as such:

`metric = load_metric('precision', average='weighted')`

Would someone kindly point me towards whatever I'm doing wrong? I'm able to fine-tune the pre-trained BERT model if I use 'accuracy' as the metric. But somehow it's not accepting my `average='weighted'` argument.

Comments:
Hello! Could you provide more information, for example by filling the issue template for bugs, including the code sample and versions? Thanks.

No problem. Sorry, I wasn't sure if this was a bug or my own mistake so I didn't use the bug template. Here you go:

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- PyTorch version (GPU?): 1.10.2+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- Longformer, BigBird: @ydshieh
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
1. @sgugger
2. @LysandreJik
3. @sgugger

## Information

Model I am using (Bert, XLNet ...):  bert-based-cased

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load own data set using `Dataset.from_pandas(my_data)`
2. Tokenize it with `AutoTokenizer.from_pretrained('bert-base-cased')`
3. Create training arguments, metric, and Trainer object to start training.

```
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=24)
training_args = TrainingArguments(output_dir='./checkpoints/my_model', evaluation_strategy="epoch")

metric = load_metric('precision', average='weighted')
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(model=model,
                  args=training_args,
                  train_dataset=train_dataset,
                  eval_dataset=eval_dataset,
                  compute_metrics=compute_metrics)
trainer.train()
```
and I got this error:
```python
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-46-5aef28bcb00d> in <module>
     14                   eval_dataset=eval_dataset,
     15                   compute_metrics=compute_metrics)
---> 16 trainer.train()

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1488 
   1489             self.control = self.callback_handler.on_epoch_end(args, self.state, self.control)
-> 1490             self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1491 
   1492             if DebugOption.TPU_METRICS_DEBUG in self.args.debug:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1600         metrics = None
   1601         if self.control.should_evaluate:
-> 1602             metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
   1603             self._report_to_hp_search(trial, epoch, metrics)
   1604 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   2262             prediction_loss_only=True if self.compute_metrics is None else None,
   2263             ignore_keys=ignore_keys,
-> 2264             metric_key_prefix=metric_key_prefix,
   2265         )
   2266 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/trainer.py in evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
   2503         # Metrics!
   2504         if self.compute_metrics is not None and all_preds is not None and all_labels is not None:
-> 2505             metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
   2506         else:
   2507             metrics = {}

<ipython-input-46-5aef28bcb00d> in compute_metrics(eval_pred)
      7     logits, labels = eval_pred
      8     predictions = np.argmax(logits, axis=-1)
----> 9     return metric.compute(predictions=predictions, references=labels)
     10 
     11 trainer = Trainer(model=model,

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/datasets/metric.py in compute(self, predictions, references, **kwargs)
    428             inputs = {input_name: self.data[input_name] for input_name in self.features}
    429             with temp_seed(self.seed):
--> 430                 output = self._compute(**inputs, **compute_kwargs)
    431 
    432             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/precision/bfadb1cf35fe89242263de7dc028b248827c08ba075659c0e812d0fc6e5237c9/precision.py in _compute(self, predictions, references, labels, pos_label, average, sample_weight)
    116     def _compute(self, predictions, references, labels=None, pos_label=1, average="binary", sample_weight=None):
    117         score = precision_score(
--> 118             references, predictions, labels=labels, pos_label=pos_label, average=average, sample_weight=sample_weight
    119         )
    120         return {"precision": float(score) if score.size == 1 else score}

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sklearn/metrics/_classification.py in precision_score(y_true, y_pred, labels, pos_label, average, sample_weight, zero_division)
   1660                                                  warn_for=('precision',),
   1661                                                  sample_weight=sample_weight,
-> 1662                                                  zero_division=zero_division)
   1663     return p
   1664 

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sklearn/metrics/_classification.py in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight, zero_division)
   1463         raise ValueError("beta should be >=0 in the F-beta score")
   1464     labels = _check_set_wise_labels(y_true, y_pred, average, labels,
-> 1465                                     pos_label)
   1466 
   1467     # Calculate tp_sum, pred_sum, true_sum ###

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sklearn/metrics/_classification.py in _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
   1294             raise ValueError("Target is %s but average='binary'. Please "
   1295                              "choose another average setting, one of %r."
-> 1296                              % (y_type, average_options))
   1297     elif pos_label not in (None, 1):
   1298         warnings.warn("Note that pos_label (set to %r) is ignored when "

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I expect the model to complete training without an error. I am able to do this if I used  `metric = load_metric('accuracy')` but not with precision, recall, or f1.
<!-- A clear and concise description of what you would expect to happen. -->


Same here

Hi there!

You are passing the ``average`` argument when you load the metric, instead, you should pass it to the `compute` method like this:

```
metric = load_metric('precision')
metric.compute(predictions=[0,1,2,3,4,4,4,4], references=[2,2,2,3,4,1,1,4], average="weighted")

Output:
----------------------
>>> {'precision': 0.625}
```

Hope this helps!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The solution of @dveni does't work if one have combined metrics.
```
metrics = evaluate.combine(['accuracy', 'f1', 'precision', 'recall'])
metrics.compute(predictions=[0,1,2,3,4,4,4,4], references=[2,2,2,3,4,1,1,4], average="weighted")
```
Currently there's not way to somehow define that when the metric used is `x` then use the `average="weighted"` otherwise don't use anything.

Even trying out something like the following will not work.
```
metrics.compute(predictions=[0,1,2,3,4,4,4,4], references=[2,2,2,3,4,1,1,4], average="weighted" if metric is not accuracy else None)
```

@kirk86  did you find a solution ? currently facing the same issue


@carlosandrea Only if you load each metric separately and then pass the argument `average='micro' or 'macro' or 'weighted'`.  In other words what @dveni  showed but you'll have to do it separately for each metric you use, e.g., `f1, precision, recall` etc.