Eval_loss in prediction is very high : transformers/examples/token-classification/run_ner.py 
Author: priyaradhakrishnan0
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 
- Tensorflow version (GPU?): not installed 
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

     

### Who can help
 examples/token-classification: @stefan-it

## Information

 I am using NER "Emerging and Rare Entities task: WNUT’17 (English NER) dataset"
I am executing the steps as prescribed in https://github.com/huggingface/transformers/tree/08ba4b4902df5a18f5ad41d9490c50fe0a4c970f/examples/token-classification

The problem arises when using prediction:
* [ ] the official example script: wnut_17.json
{
"data_dir": "/home/priya/data_wnut_17",
"labels": "/home/priya/data_wnut_17/labels.txt",
"model_name_or_path": "bert-large-cased",
"output_dir": "wnut-17-model-1",
"max_seq_length": 128,
"num_train_epochs": 3,
"per_device_train_batch_size": 16,
"save_steps": 425,
"seed": 1,
"do_train": true,
"do_eval": true,
"do_predict": true,
"fp16": false
}
* [ ] my own modified scripts: wnut_17_mod.json
{
"data_dir": "/home/priya/data_wnut_17",
"labels": "/home/priya/data_wnut_17/labels.txt",
"model_name_or_path": "bert-large-cased",
"output_dir": "wnut-17-model-1",
"max_seq_length": 128,
"num_train_epochs": 3,
"per_device_train_batch_size": 16,
"save_steps": 425,
"seed": 1,
"do_train": **false,**
"do_eval": **false**,
"do_predict": true,
"fp16": false,
"overwrite_output_dir":false
}


The tasks I am working on is:
* [ ] re-run WNUT’17 dataset. 

My end-goal is to identify abbreviation and explanation from sentences (labels B-abbr, I-abbr, B-expl, I-expl and O). For the example sentence

> Here GAAP stands for Generally accepted accounting principles

we should get token classified as

> Here O
> GAAP B-abbr
> stands O
> for O
> Generally B-expl
> accepted I-expl
> accounting I-expl
> principles I-expl

## To reproduce

Steps to reproduce the behavior:

1.python run_ner.py wnut_17.json
Prediction: 100%|████████████████████████████████████████████████████████| 162/162 [01:08<00:00,  2.38it/s]
10/06/2020 07:21:15 - INFO - __main__ -     eval_loss = 0.2851179020827574
10/06/2020 07:21:15 - INFO - __main__ -     eval_accuracy_score = 0.9511413182867402
10/06/2020 07:21:15 - INFO - __main__ -     eval_precision = 0.5997392438070405
10/06/2020 07:21:15 - INFO - __main__ -     eval_recall = 0.4263206672845227
10/06/2020 07:21:15 - INFO - __main__ -     eval_f1 = 0.49837486457204777

2.python run_ner.py wnut_17_mod.json
Prediction: 100%|████████████████████████████████████████████████████████| 162/162 [01:08<00:00,  2.38it/s]
/opt/conda/lib/python3.7/site-packages/transformers/trainer.py:1175: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
  warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)
10/06/2020 08:30:41 - INFO - __main__ -     eval_loss = 2.827890293463151
10/06/2020 08:30:41 - INFO - __main__ -     eval_accuracy_score = 0.016072497221509788
10/06/2020 08:30:41 - INFO - __main__ -     eval_precision = 0.0065180614986565825
10/06/2020 08:30:41 - INFO - __main__ -     eval_recall = 0.12140871177015755
10/06/2020 08:30:41 - INFO - __main__ -     eval_f1 = 0.012371912924399112

## Expected behavior
I am seeing a 10-fold increase in eval-loss from 0.28 to 2.8. 
Other than the changes in  wnut_17_mod.json , I have done no other changes. Please advice how to achieve the published eval_loss and performance.

Thanks,


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any updates?!? On training, it is very low and on the evaluation its insanely high, I tried to check if the hyper parameters were the problem, but I didn't find anything.

No. The eval_loss is low. No updates.

I was reading more about the metrics and it looks like that for a specific task they place a specific meaning, e.g I'm doing multi-label classification so these are the [metrics](https://simpletransformers.ai/docs/classification-models/#evaluating-a-classification-model):

**LRAP**

Label ranking average precision.

Label ranking average precision (LRAP) is the average over each ground truth label assigned to each sample, of the ratio of true vs. total labels with lower score.

The obtained score is always strictly greater than 0 and the best value is 1.

**Evaluation Loss**

Binary Cross Entropy Loss.

It is a Sigmoid activation plus a Cross-Entropy loss. Unlike Softmax loss it is independent for each vector component (class), meaning that the loss computed for every NN output vector component is not affected by other component values.

Cross-entropy loss awards lower loss to predictions which are closer to the class label. The accuracy, on the other hand, is a binary true/false for a particular sample. That is, Loss here is a continuous variable i.e. it's best when predictions are close to 1 (for true labels) and close to 0 (for false ones).

Theoretically, the output of the model is not wrong, but the interpretation is.

- References

[Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and all those confusing names](https://gombru.github.io/2018/05/23/cross_entropy_loss/#:~:text=is%20available%20here-,Binary%20Cross%2DEntropy%20Loss,affected%20by%20other%20component%20values.)

[Loss vs Accuracy](https://kharshit.github.io/blog/2018/12/07/loss-vs-accuracy#:~:text=Cross%2Dentropy%20loss%20awards%20lower,0%20(for%20false%20ones).)

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.