Your example code for WNUT NER produces array indexing ValueError
Author: githubrandomuser2017
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@stefan-it, @sgugger
## Information

Model I am using (Bert, XLNet ...): DistilBERT

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

I'm trying to run the example code Advanced Guides --> Fine-tuning with custom datasets --> [Token Classification with W-NUT Emerging Entities](https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities).

Steps to reproduce the behavior:

1. I already have a [Google CoLab notebook with your code](https://colab.research.google.com/drive/1i5N7Xc-i91bqXmcp9hamt5q3a_5a-ZnZ?usp=sharing).
2. I use the `tokenizer` with `max_length=64`, which is typically my "best practice" choice. Note that if I set `max_length=None`, everything runs successfully.
```python
max_length = 64
encodings = tokenizer(texts, is_split_into_words=True, max_length=max_length, return_offsets_mapping=True, padding=True, truncation=True)
```
3. When I run `encode_tags()` on the WNUT data, I get a ValueError
```python
labels = encode_tags(tags, encodings)
     11         # set labels whose first offset position is 0 and the second is not 0
---> 12         doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels
     13         encoded_labels.append(doc_enc_labels.tolist())
     14 

ValueError: NumPy boolean array indexing assignment cannot assign 29 input values to the 24 output values where the mask is true
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I expect that `encode_tags()` should return the correct IOB tag labels when I run your `Tokenizer` with a `max_length=64`.

Comments:
Hi,
not a HuggingFace developer but I came across the same problem. I think this is this is due to the fact that the Tokenizer is truncating sequences longer than 64 so there is a mismatch in length between `tags` and `encodings`. This is also why it's fixed when you increase the max_lenght. Another reason may be that some characters in your sentences are not properly decoded because of wrong charset detection. I hope this helps.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am also facing this issue. I am using custom dataset and haven't passed any max_length argument to the tokenizer. 

Any idea how to fix this ? But same piece of code works well on W-NUT dataset

> Hi,
> not a HuggingFace developer but I came across the same problem. I think this is this is due to the fact that the Tokenizer is truncating sequences longer than 64 so there is a mismatch in length between `tags` and `encodings`. This is also why it's fixed when you increase the max_lenght. Another reason may be that some characters in your sentences are not properly decoded because of wrong charset detection. I hope this helps.

I observed that in the notebook shared by Hugging face for W-Nut dataset either, the tags and encodings length (for each record) are not same. So hoping that shouldn't be the issue. 

@joeddav I am facing the same issue when switching to another dataset, what could be the problem? the behavior continues even with  setting `max_length=None`

For me the error occurred using the example code in combination with a sentence piece tokenizer (e.g. XLM-RoBERTa). Switching to the updated code used in the run_ner.py script (https://github.com/huggingface/transformers/blob/ad072e852816cd32547504c2eb018995550b126a/examples/token-classification/run_ner.py) solved the issue for me. 

I figured out the problem. A typical input instance has `N` tokens and `N` NER tags with a one-to-one correspondence. When you pass in the sentence to the tokenizer, it will add `k` more tokens for either (1) subword tokens (e.g. `##ing`) or (2) special model-specific tokens (e.g. `[CLS]` or `[SEP]`. So now you have `N+k` tokens and `N` NER tags.

If you apply a max length truncation (e.g. `64`), then those `N+k` tokens will get truncated to `64`, leaving an unpredictable mix of valid tokens and special tokens because both types of tokens may have been truncated. However, there are still `N` NER tags which may not match up against valid tokens because the latter may have been truncated.

I fixed the problem by one of several approaches:

1. Removing data instances that are problematically long. For example, I removed sentences which had more than 45 tokens. Using Pandas really help out here.
2. Increasing the truncation length to, say, 128, or whatever number that's longer than any `N+k`. However, this increase forces me to reduce my batch size due to GPU memory constraints.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

I solved the issue by replacing
```python
doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels
encoded_labels.append(doc_enc_labels.tolist())
```

with

```python
mask = (arr_offset[:, 0] == 0) & (arr_offset[:, 1] != 0)
doc_enc_labels[mask] = doc_labels[:np.sum(mask)]
encoded_labels.append(doc_enc_labels.tolist())
```

By this way, it will only map the first `np.sum(mask)` true indices of `doc_labels` in case of any indexing problem. I am a newbie 🤗 Transformers user, and I wonder if this solution may cause any problems.

I have this same issue but 
`mask = (arr_offset[:, 0] == 0) & (arr_offset[:, 1] != 0)
doc_enc_labels[mask] = doc_labels[:np.sum(mask)]
encoded_labels.append(doc_enc_labels.tolist())
`
did not work after the first encoded_labels run

Guys if the example has issues, why even put it out there and have us chaise our tails?

Hey! The example is currently being rewritten here by @stevhliu: https://github.com/huggingface/transformers/pull/13923

@LysandreJik Thanks for revisiting this problem. I feel that aligning tokens, token labels, and sub-world pieces is too complex for users of the library to implement themselves. Can you (HuggingFace) please provide some utility functions to make this task easier?

Hi @githubrandomuser2017, the examples we provide showcase exactly how to do that, for example here: https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py#L370-L404

Does this utility function help you out? 

PR #13923 was merged with the new version of this example. Closing this issue, feel free to reopen/comment if the issue arises again.

@LysandreJik 
> Hi @githubrandomuser2017, the examples we provide showcase exactly how to do that, for example here: https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py#L370-L404
> 
> Does this utility function help you out?

I'll let other users chime in.

Even though this issue was closed, some old code still use the old version of `encode_tags` method. From the example [above](https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py#L370-L404), we found the hugging face [page](https://huggingface.co/learn/nlp-course/en/chapter7/2?fw=pt#processing-the-data) that explains the `align_labels_with_tokens` method. It is a more robust approach to any unintentional words that are removed by fast tokenizers in the normalization process.