Added missing code in exemplary notebook - custom datasets fine-tuning
Author: Pawloch247
Pull request: yes

# What does this PR do?

Added missing code in tokenize_and_align_labels function in the exemplary notebook on custom datasets - token classification.
The missing code concerns adding labels for all but the first token in a single word.
The added code was taken directly from huggingface official example - this [colab notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb).

Fixes # (issue)

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- maintained examples (not research project or legacy): @sgugger, @patil-suraj



Comments:
_The documentation is not available anymore as the PR was closed or merged._

I don't understand what you mean, the notebook has the same code as the example: they are automatically synced at each merge in the Transformers repo.

You're right. I pasted the wrong link so the comparison made no sense. The link should have been to the colab notebook: 
https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb

In the colab notebook `tokenize_and_align_labels` has the `else` clause which is missing in the notebook in Github and hence is missing here: https://huggingface.co/docs/transformers/custom_datasets#token-classification-with-wnut-emerging-entities

Those are two different tutorials, it's normal they have different code. The one in the main documentation is left as simple as possible on purpose.

In the main documentation, there is a mention about "Only labeling the first token of a given word. Assign -100 to the other subtokens from the same word.". However, it is not the case with the code below, the `tokenize_and_align_labels` function, where not only are the other subtokens assigned with the true labels but also `previous_word_idx` is not being updated. Such contradiction was ambiguous for me. Only after having dug deeper (the colab notebook), I understood that this part of code was missing from the official documentation. I do not think that these few lines were omitted on purpose. If you don't think it makes such a difference, close this pr (or let me know if I should be the one to close it).

I adjusted your comments. I guess I could have been more precise from the beginning. Thanks.