[DOC] Fine-Tuning NER Custom Dataset Clarification
Author: jorahn
Pull request: no

I'm following [this](https://huggingface.co/transformers/custom_datasets.html#tok-ner) guide for fine-tuning for NER with a custom dataset. I struggled with the example code for `def encode_tags()` until I realized, that the tokens per sample are limited to 512 and my dataset exceeded this in some instances. This resulted in errors like this:
`ValueError: NumPy boolean array indexing assignment cannot assign 544 input values to the 464 output values where the mask is true`.

I currently assume, the limit is due to the specific Tokenizer. I'm using `tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-cased')` as in the example. 

I'm proposing to add a clarification about the token limit per sample assumption like this:

https://github.com/huggingface/transformers/edit/master/docs/source/custom_datasets.rst
Line 365 and following:
> Let's write a function to do this. This is where we will use the ``offset_mapping`` from the tokenizer as mentioned
> above. For each sub-token returned by the tokenizer, the offset mapping gives us a tuple indicating the sub-token's
> start position and end position relative to the original token it was split from. That means that if the first position
> in the tuple is anything other than ``0``, we will set its corresponding label to ``-100``. While we're at it, we can
> also set labels to ``-100`` if the second position of the offset mapping is ``0``, since this means it must be a
> special token like ``[PAD]`` or ``[CLS]``.
And append: `Be aware that this example has an upper limit of 512 tokens per sample.`

Let me know your thoughts and I'll open a PR, if you find this useful.

Comments:
When did you encounter that `ValueError`? Normally, if you pass text into the `__call__` method of a tokenizer with `truncation` set to `True`, you shouldn't encounter any errors, as sequences that are too long are truncated.

Hi Niels, thanks for your feedback. The Exception is raised when calling `encode_tags`. I'm following the tutorial code, just my dataset is not WNUT-17.
> train_labels = encode_tags(train_tags, train_encodings)
> val_labels = encode_tags(val_tags, val_encodings)

Update: Even when I ensure the number of tokens per sample is <= 512, I get ValueErrors from calling `encode_tags` on some samples. I'll try to understand this better or provide a demo.

Update: The function `tokenize_and_align_labels` from the [token classification example notebook](https://github.com/huggingface/notebooks/blob/master/examples/token_classification.ipynb) (cell 23) works fine on my data.

Having the same error on my custom dataset, with truncation set to true, with the Camembert tokenizer.  
`ValueError: NumPy boolean array indexing assignment cannot assign 1078 input values to the 319 output values where the mask is true`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Having the same issue, from the same tutorial, with a custom dataset. Any ideas on how to fix it?

@Famaral97 You may want to take a look here: https://github.com/huggingface/transformers/issues/11652#issuecomment-839496635 
This worked well for my dataset.

@jorahn Thanks for letting us know about the tokenize_and_align_labels() function. But when I follow the method mentioned in the notebook I'm getting an error with data collator saying:
AttributeError: 'tokenizers.Encoding' object has no attribute 'keys'