Repository: huggingface/transformers

Issue:
run_ner.py with xlm-roberta-base raises an IndexError in tokenize_and_align_labels
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
 `transformers` version: 4.0.0  (and the example scripts from git master aka 72d6c9c6)
- Platform: Linux-4.19.0-12-amd64-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: True (but we don’ŧ get that far)
- Using distributed or parallel set-up in script?: False

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
git blame says @sgugger 

## Information

Model I am using (Bert, XLNet ...): xlm-roberta-base

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `python3 run_ner.py --model_name_or_path xlm-roberta-base --task_name ner --dataset_name conll2003 --label_all_tokens --do_train --do_eval --output_dir finetuning-output`

Crashes with the following stacktrace:

```
Traceback (most recent call last):
  File "run_ner.py", line 394, in <module>
    main()
  File "run_ner.py", line 292, in main
    tokenized_datasets = datasets.map(
  File "/home/vitt/.conda/envs/cuda/lib/python3.8/site-packages/datasets/dataset_dict.py", line 286, in map
    {
  File "/home/vitt/.conda/envs/cuda/lib/python3.8/site-packages/datasets/dataset_dict.py", line 287, in <dictcomp>
    k: dataset.map(
  File "/home/vitt/.conda/envs/cuda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1239, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/home/vitt/.conda/envs/cuda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1210, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "run_ner.py", line 277, in tokenize_and_align_labels
    current_label = label_to_id[label[label_index]]
IndexError: list index out of range
```

From a little debugging, the problem seems to be that this code assumes there are only as many sequences with `offset[0] == 0 and offset[1] != 0` as there are words in the original input (and thus as there are labels):

https://github.com/huggingface/transformers/blob/72d6c9c68ba19b2e991b0d7a32989410399b33f5/examples/token-classification/run_ner.py#L276-L278

However, the SentencePiece tokenizer may split input words to sequences starting with a single `'▁'` token. Then, the offset mapping for '▁' will be `(0, 1)` and for the following token `(0, x)` (E.g. '.' in the CONLL data ⇒ `['▁', '.']` with offsets `[(0, 1), (0, 1)]` or ['NACCO'] ⇒ `('▁', (0, 1)), ('NAC', (0, 3)), ('CO', (3, 5))`.

(Could this use `tokenized_inputs.word_ids()` instead?)


Discussion:

Resolving patch:
diff --git a/examples/token-classification/run_ner.py b/examples/token-classification/run_ner.py
--- a/examples/token-classification/run_ner.py
+++ b/examples/token-classification/run_ner.py
@@ -35,6 +35,7 @@
     AutoTokenizer,
     DataCollatorForTokenClassification,
     HfArgumentParser,
+    PreTrainedTokenizerFast,
     Trainer,
     TrainingArguments,
     set_seed,
@@ -250,6 +251,14 @@ def get_label_list(labels):
         cache_dir=model_args.cache_dir,
     )
 
+    # Tokenizer check: this script requires a fast tokenizer.
+    if not isinstance(tokenizer, PreTrainedTokenizerFast):
+        raise ValueError(
+            "This example script only works for models that have a fast tokenizer. Checkout the big table of models "
+            "at https://huggingface.co/transformers/index.html#bigtable to find the model types that meet this "
+            "requirement"
+        )
+
     # Preprocessing the dataset
     # Padding strategy
     padding = "max_length" if data_args.pad_to_max_length else False
@@ -262,28 +271,25 @@ def tokenize_and_align_labels(examples):
             truncation=True,
             # We use this argument because the texts in our dataset are lists of words (with a label for each word).
             is_split_into_words=True,
-            return_offsets_mapping=True,
         )
-        offset_mappings = tokenized_inputs.pop("offset_mapping")
         labels = []
-        for label, offset_mapping in zip(examples[label_column_name], offset_mappings):
-            label_index = 0
-            current_label = -100
+        for i, label in enumerate(examples[label_column_name]):
+            word_ids = tokenized_inputs.word_ids(batch_index=i)
+            previous_word_idx = None
             label_ids = []
-            for offset in offset_mapping:
-                # We set the label for the first token of each word. Special characters will have an offset of (0, 0)
-                # so the test ignores them.
-                if offset[0] == 0 and offset[1] != 0:
-                    current_label = label_to_id[label[label_index]]
-                    label_index += 1
-                    label_ids.append(current_label)
-                # For special tokens, we set the label to -100 so it's automatically ignored in the loss function.
-                elif offset[0] == 0 and offset[1] == 0:
+            for word_idx in word_ids:
+                # Special tokens have a word id that is None. We set the label to -100 so they are automatically
+                # ignored in the loss function.
+                if word_idx is None:
                     label_ids.append(-100)
+                # We set the label for the first token of each word.
+                elif word_idx != previous_word_idx:
+                    label_ids.append(label_to_id[label[word_idx]])
                 # For the other tokens in a word, we set the label to either the current label or -100, depending on
                 # the label_all_tokens flag.
                 else:
-                    label_ids.append(current_label if data_args.label_all_tokens else -100)
+                    label_ids.append(label_to_id[label[word_idx]] if data_args.label_all_tokens else -100)
+                previous_word_idx = word_idx
 
             labels.append(label_ids)
         tokenized_inputs["labels"] = labels


Test patch: