run_tf_ner.py doesn't work with unlabelled test data
Author: 0dust
Pull request: no
Labels: name: wontfix

When running `run_tf_ner.py` in `predict` mode if all the labels in test data are `O`, script errors out with 
`  File "/home/himanshu/.local/lib/python3.7/site-packages/numpy/lib/function_base.py", line 423, in average
    "Weights sum to zero, can't be normalized")
ZeroDivisionError: Weights sum to zero, can't be normalized
`
This is because `pad_token_label_id` https://github.com/huggingface/transformers/blob/cae334c43c49aa770d9dac1ee48319679ee8c72c/examples/ner/run_tf_ner.py#L511 , `label_id` for `O` are both zero, resulting in empty `y_pred`
https://github.com/huggingface/transformers/blob/cae334c43c49aa770d9dac1ee48319679ee8c72c/examples/ner/run_tf_ner.py#L364-L367 Shouldn't the `pad_token_label_id` be different?

Comments:
I have noticed the same issue and posted a question here: https://stackoverflow.com/questions/60732509/label-handling-confusion-in-run-tf-ner-example

I think `pad_token_label_id` should definitely not fall into the range of actual labels. Maybe we can make it `-1` or `num(label)` or something. Also as shown in `convert_examples_to_features()`, `pad_token_label_id` is not only used for pad tokens at the end of the sequence, but also for non-first tokens inside a word when the word is split up to multiple tokens. Accordingly, during prediction, only the label of the first token in each word is used. So I am wondering if we should modify `input_mask` so that the loss does not take into account non-first tokens in a word. 

I tried to set `pad_token_label_id = -1`, mask out non-first tokens in each word by changing `input_mask`, and change `num_labels` to `len(labels)` instead of `len(labels) + 1`. The training and evaluation can run, but the F1-score on the test set becomes much lower (on both conll03 English and Ontonotes English). I am still confused about this.

I also found the issue, `pad_token_label_id = 0` and first labels id also 0, seems a bug. @jplu 

Hey !

The "fake" pad token id must be 0 and the first "true" token id must be 1. This is important to make the network able to make the difference between a padding and a word that is not part of an entity.

I just have tested the script on multiple NER datasets and works perfectly without any change, so I think, if there is an issue it is only with unlabeled data.

@0dust: the exception you mentioned do not rely on where the script has failed, I will try with testing on unlabeled data to see if I get the same issue. To be honest when I developed this script I never tried with unlabeled testing data.

@VDCN12593: -1 doesn't work because TF do not take into account negative ids.

@jplu Exactly! That was just a short desription I could think of.

@0dust Sorry I cannot reproduce your issue, everything works fine for me even with unlabeled data... Please try to reproduce the example over germeval in the README by removing the label column in the test file. For me it works like expected.

If you still get the same issue, please provide an example of data for which it doesn't works :)

@jplu Thank you for your reply! Here's my thoughts

1. In `run_tf_ner`, it doesn't make the "true" labels start from 1, so that's definitely a bug.

2. We can make `pad_token_label_id = -1`, as long as we also mask out all the non-first tokens inside each word (in function `convert_examples_to_features()`) so that there softmax output are not taken by the loss. I think this makes more sense because we only use the first token (wordpiece) of each word for the tagging, and don't care about the output of the other tokens. This method is also supported by some people like in here: https://www.vamvas.ch/bert-for-ner/

@VDCN12593 thanks for your feedback! Indeed the script to generate the feature has changed since I make the script and I did not update it to follow the changes, so yes your first bullet point is true since then. About the second, yes I know that but keeping all the values without ignoring some makes your model converging faster.

Anyway, I know this script is unusual and difficult to properly follow. So, this weekend, once I have some time, I will fully review it and make it much easier to understand. I will let you know once done.

I have done all the changes that was raising some confusion in this PR https://github.com/huggingface/transformers/pull/3511. Basically, the pad token label id to -1 and removing the softmax. The training is a bit longer and results stay unchanged.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.