New tokenizers issue in NER demo
Author: srush
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: Should Fix

# 🐛 Bug

When I use transformers 2.5 I get the following error when running the run_ner demo. It seems to work from when I use 2.4. I am guessing this is because of a slight difference in the tokenization script? It seems to fail even when running with the base German run.sh in the ner/ directory. 

```Traceback (most recent call last):
  File "run_pl_ner.py", line 233, in <module>
    trainer = generic_train(model, args)
  File "/content/transformers/examples/ner/transformer_base.py", line 268, in generic_train
    trainer.fit(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 911, in fit
    self.single_gpu_train(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/distrib_parts.py", line 464, in single_gpu_train
    self.optimizers, self.lr_schedulers = self.init_optimizers(model.configure_optimizers())
  File "/content/transformers/examples/ner/transformer_base.py", line 92, in configure_optimizers
    * float(self.hparams.num_train_epochs)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/core/decorators.py", line 19, in _get_data_loader
    value = fn(self)  # Lazy evaluation, done only once.
  File "/content/transformers/examples/ner/transformer_base.py", line 132, in train_dataloader
    return self.load_dataset("train", self.hparams.train_batch_size)
  File "run_pl_ner.py", line 50, in load_dataset
    dataset = self.load_and_cache_examples(labels, self.pad_token_label_id, mode)
  File "run_pl_ner.py", line 175, in load_and_cache_examples
    pad_token_label_id=pad_token_label_id,
  File "/content/transformers/examples/ner/utils_ner.py", line 182, in convert_examples_to_features
    assert len(label_ids) == max_seq_length
```

Comments:
This might be https://github.com/huggingface/transformers/issues/2917

@srush I'm looking at it right now 

I have the same issue while running run_tf_ner.py on the German NER dataset. I got the same AssertionError as below:
>  File "run_tf_ner.py", line 655, in <module>
>     app.run(main)
>   File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
>     _run_main(main, args)
>   File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
>     sys.exit(main(argv))
>   File "run_tf_ner.py", line 540, in main
>     args, tokenizer, labels, pad_token_label_id, train_batch_size, mode="train"
>   File "run_tf_ner.py", line 451, in load_and_cache_examples
>     pad_token_label_id=pad_token_label_id,
>   File "/content/transformers/examples/ner/utils_ner.py", line 182, in convert_examples_to_features
>     assert len(label_ids) == max_seq_length
> AssertionError

My idea is that the [pad_token_label_id = 0](https://github.com/huggingface/transformers/blob/94ff2d6ee8280c5595b92c1128c0f18e44925e56/examples/ner/run_tf_ner.py#L511)  may conflict with the orginal label_list id. Becase in utils_ner.py (line 104), `label_map = {label: i for i, label in enumerate(label_list)}` . 
By the way, I run the same code on CoNLL-2003 dataset with default labels:
["O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"]
No such error message... It may be 'O' is the first token but in Germen dataset 'O' was the last label in label.text.  
I don't know if this is the real problem, but I hope this bug can be fixed soon. Thx.
 


> @srush I'm looking at it right now

I have the same issue as @srush. Any idea yet what the issue could be?

It works fine with 2.4 so it is likely an issue with the new tokenizer.

Hi @srush ,

I am using transformers==2.4.1, but still facing the problem with a custom data set (with extra labels).  It is working with the german dataset though.  Could you be more specific about the version of transformers that you use.
Thanks


Hi @cibinjohn , 

Can you tell us which tokenizers / model you're using ? We fixed something for `bert-base-multilingual` should be merge in master quite soon.


Any one got solution for this yet? I have the same issue with this. I used transformer 2.5.1 and the latest tokenizer come with the transformers installation by default. 
Looks like the assert len(label_ids) == max_seq_length because len(label_ids) is one more than max_seq_length while 3 other asserts before it pass the tests. 

@yuyongze  Have you made any progress on this?

I think `pad_token_label_id = 0` is actually another issue: #3332 
And related: https://stackoverflow.com/questions/60732509/label-handling-confusion-in-run-tf-ner-example

Experiencing the same problem launching run_ner.py on the WNUT17 dataset, although on German and CONLL-2003 everything works fine.

I just want to run the ner demo (TF-version) but the same issue/ error raises.. Tried with transformers version 2.4/5/6 still the same error raises. Has anyone a solution? 

Edit: PyTorch script seems to work

@mfuntowicz I am using `bert-base-multilingual` model & transformers==2.4.1.  

Could you let me know when the solution will be merged with master.  Looking forward to hear from you.
Thanks in advance



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.