Issue with 'char_to_token()' function of DistilBertTokenizerFast 
Author: PremalMatalia
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.1
- Platform: Google Colab
- Python version: 3.8
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: NA

### Who can help:   **tokenizers: @mfuntowicz**

## Information

Model I am using DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased') to tokenize Squad 2.0 train and validate dataset. 

The problem arises when using below code snippet to add_token_positions (start and end position) as below from https://huggingface.co/transformers/custom_datasets.html:

_def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_positions.append(**encodings.char_to_token(i, answers[i]['answer_start'])**)
        end_positions.append(**encodings.char_to_token(i, answers[i]['answer_end'] - 1**))
        # if None, the answer passage has been truncated
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)_




The tasks I am working on is:
*Training model on SQUaD 2.0 using code given on https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0

## To reproduce

Steps to reproduce the behavior:

1. Follow the steps given on https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0 and then verify start and end position outcome using below code snippet in Expected behavior


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior:
- Start and End position are being defined using above code snippet which will be provided as training/validation data to model but end position is not derived as correct value due to some issue with char_to_token() function which is used to find out end position.
- Please find below snippet for verification that answer using start and end position after tokenization is not matching with actual answer.
- So the training data which is being fed to model after tokenization is incorrect

idx=8
print(f'Actual context: {train_contexts[idx]}')
print(f'Actual question: {train_questions[idx]}')
print(f"Actual answer: {train_answers[idx]['text']}")

start_position=train_encodings.char_to_token(idx,train_answers[idx]['answer_start'])
end_position =train_encodings.char_to_token(idx,train_answers[idx]['answer_end'])
print(f"Answer after tokenization: {tokenizer.convert_ids_to_tokens(train_encodings['input_ids'][idx][start_position:end_position])}") 

OUTPUT:
**Actual context:** Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".
**Actual question:** When did Beyoncé rise to fame?
**Actual answer:** late 1990s
**Answer after tokenization:** ['late', '1990s', 'as', 'lead', 'singer', 'of', 'r', '&', 'b', 'girl', '-', 'group', 'destiny', "'", 's', 'child', '.', 'managed', 'by', 'her', 'father', ',', 'mathew', 'knowles', ',', 'the', 'group', 'became', 'one', 'of', 'the', 'world', "'", 's', 'best', '-', 'selling', 'girl', 'groups', 'of', 'all', 'time', '.', 'their', 'hiatus', 'saw', 'the', 'release', 'of', 'beyonce', "'", 's', 'debut', 'album', ',', 'dangerously', 'in', 'love', '(', '2003', ')', ',', 'which', 'established', 'her', 'as', 'a', 'solo', 'artist', 'worldwide', ',', 'earned', 'five', 'grammy', 'awards', 'and', 'featured', 'the', 'billboard', 'hot', '100', 'number', '-', 'one', 'singles', '"', 'crazy', 'in', 'love', '"', 'and', '"', 'baby', 'boy', '"', '.', '[SEP]', 'when', 'did', 'beyonce', 'rise', 'to', 'fame', '?', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']

Comments:
Hey @PremalMatalia, 

Could you please provide a copy/paste ready code-snippet that can be used to reproduce the error. By copy/past ready code snippet I mean something like:

```python
from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')

# ... add all the necessary code here to be able to reproduce your error
```

. Thanks! 

Hello Patrick,
Please find entire code starting from SQuAD 2.0 training data download to encoding to adding start and end position as below:

```python
!pip install wget
!pip install transformers==4.0.1

import wget
import json
from pathlib import Path
import os
import json
from transformers import DistilBertTokenizerFast,TFDistilBertForQuestionAnswering
import tensorflow as tf

# Import training data
!mkdir squad
train_source = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json'
train_dest  = 'squad/train-v2.0.json'
wget.download(train_source,train_dest)

## Function to extract context, questions and answers
def read_squad(path,dataset='train'):
   
    contexts  = []
    questions = []
    question_ids = []
    answers   = []
    blank_counter = 0
    append_flag = False

    with open(path) as f:
        data = json.load(f)
    
    ## Loop over entire dataset
    for article_id in range(len(data['data'])):
        paragraphs = data['data'][article_id]['paragraphs']
        ## Loop over all the paragraphs
        for paragraph in paragraphs:
            context   = paragraph['context']
            qas       = paragraph['qas']
            ## Loop over Questions and Answers
            for qa in qas:
                append_flag=False
                question = qa['question']
                question_id = qa['id']
                ## Select 1st answer if answers are available 
                if qa['answers']:
                    answer = qa['answers'][0]
                    append_flag = True
                    ## Append contexts and questions in a list and answers in a list as dictionary
                    contexts.append(context)
                    questions.append(question)
                    question_ids.append(question_id)
                    answers.append(answer) 
                
    return contexts, questions, question_ids, answers

train_contexts, train_questions,_, train_answers = read_squad('squad/train-v2.0.json')


## Function to update answer_start and answer_end 
def add_end_idx(answers, contexts):
    '''
    Description: This function is to find out character position at which the answer ends in the passage. 
                 Also corrects answer start and end position if the SQuAD answers are off by one or two characters
    Input: List of all answers, List of all contexts
    Output: Updated list with answer end position
    '''
    for answer, context in zip(answers, contexts):
        # Your code here
        if answer['answer_start'] is None:
            answer['answer_end'] = None
        else:
            answer_text  = answer['text']
            answer_start = answer['answer_start']
            answer_end   = len(answer_text) + answer_start

            #Sometimes answers are off by a character or two
            if context[answer_start:answer_end] == answer['text']:
                answer['answer_end'] = answer_end
            # If the answer text is off by 1 character
            elif context[answer_start-1:answer_end-1] == answer_text:
                answer['answer_start'] = answer_start - 1
                answer['answer_end'] = answer_end - 1     
            # If the answer text is off by 2 characters
            elif context[answer_start-2:answer_end-2] == answer_text:
                answer['answer_start'] = answer_start - 2
                answer['answer_end'] = answer_end - 2     

add_end_idx(train_answers, train_contexts)

## Tokenize training data
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True,
                                          return_offsets_mapping=True,
                                          return_overflowing_tokens=True)

## Find out start_position and end_position in encoded dataset
def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []

    for idx in range(len(answers)):
        start_positions.append(encodings.char_to_token(idx, answers[idx]['answer_start']))
        if answers[idx]['answer_end'] is None:
          end_positions.append(encodings.char_to_token(idx, answers[idx]['answer_end']))
        else:
          end_positions.append(encodings.char_to_token(idx, answers[idx]['answer_end'] - 1))
        
        #if None, the answer passage has been truncated due to words > 512 so setting last position as 511
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length-1
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length-1
    
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

add_token_positions(train_encodings, train_answers)

## Validate answers based on start_position and end_position with actual answer for some random index
idx=8
print(f'Actual context: {train_contexts[idx]}')
print(f'Actual question: {train_questions[idx]}')
print(f"Actual answer: {train_answers[idx]['text']}")

start_position=train_encodings.char_to_token(idx,train_answers[idx]['answer_start'])
end_position =train_encodings.char_to_token(idx,train_answers[idx]['answer_end'])

## ******This shows how start_position and end_position derived by using char_to_token() function is not correct******
print(f"Answer after tokenization: {tokenizer.convert_ids_to_tokens(train_encodings['input_ids'][idx][start_position:end_position])}")
```

Completely agree with @PremalMatalia. 
The problem is with -
start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))

We are getting None where as we should have got start token position. 
char_to_token is not able to convert from string position to token position.

Thanks @PremalMatalia - I think I can reproduce.

The PR attached below should fix the problem. Can you check it again with the proposed fix?

Thanks @patrickvonplaten for quick action.
If I understand correctly, fix has been merged to original char_to_token() function? If yes, we can directly use the same function without any changes in code from myside. Is that correct?

No the `char_to_token()` function was always correct (It's actually a rust function from tokenizers that is used with python bindings). The function was simply used incorrectly, so I updated the docs.

> Hello Patrick,
> Please find entire code starting from SQuAD 2.0 training data download to encoding to adding start and end position as below:
> 
> ```python
> !pip install wget
> !pip install transformers==4.0.1
> 
> import wget
> import json
> from pathlib import Path
> import os
> import json
> from transformers import DistilBertTokenizerFast,TFDistilBertForQuestionAnswering
> import tensorflow as tf
> 
> # Import training data
> !mkdir squad
> train_source = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json'
> train_dest  = 'squad/train-v2.0.json'
> wget.download(train_source,train_dest)
> 
> ## Function to extract context, questions and answers
> def read_squad(path,dataset='train'):
>    
>     contexts  = []
>     questions = []
>     question_ids = []
>     answers   = []
>     blank_counter = 0
>     append_flag = False
> 
>     with open(path) as f:
>         data = json.load(f)
>     
>     ## Loop over entire dataset
>     for article_id in range(len(data['data'])):
>         paragraphs = data['data'][article_id]['paragraphs']
>         ## Loop over all the paragraphs
>         for paragraph in paragraphs:
>             context   = paragraph['context']
>             qas       = paragraph['qas']
>             ## Loop over Questions and Answers
>             for qa in qas:
>                 append_flag=False
>                 question = qa['question']
>                 question_id = qa['id']
>                 ## Select 1st answer if answers are available 
>                 if qa['answers']:
>                     answer = qa['answers'][0]
>                     append_flag = True
>                     ## Append contexts and questions in a list and answers in a list as dictionary
>                     contexts.append(context)
>                     questions.append(question)
>                     question_ids.append(question_id)
>                     answers.append(answer) 
>                 
>     return contexts, questions, question_ids, answers
> 
> train_contexts, train_questions,_, train_answers = read_squad('squad/train-v2.0.json')
> 
> 
> ## Function to update answer_start and answer_end 
> def add_end_idx(answers, contexts):
>     '''
>     Description: This function is to find out character position at which the answer ends in the passage. 
>                  Also corrects answer start and end position if the SQuAD answers are off by one or two characters
>     Input: List of all answers, List of all contexts
>     Output: Updated list with answer end position
>     '''
>     for answer, context in zip(answers, contexts):
>         # Your code here
>         if answer['answer_start'] is None:
>             answer['answer_end'] = None
>         else:
>             answer_text  = answer['text']
>             answer_start = answer['answer_start']
>             answer_end   = len(answer_text) + answer_start
> 
>             #Sometimes answers are off by a character or two
>             if context[answer_start:answer_end] == answer['text']:
>                 answer['answer_end'] = answer_end
>             # If the answer text is off by 1 character
>             elif context[answer_start-1:answer_end-1] == answer_text:
>                 answer['answer_start'] = answer_start - 1
>                 answer['answer_end'] = answer_end - 1     
>             # If the answer text is off by 2 characters
>             elif context[answer_start-2:answer_end-2] == answer_text:
>                 answer['answer_start'] = answer_start - 2
>                 answer['answer_end'] = answer_end - 2     
> 
> add_end_idx(train_answers, train_contexts)
> 
> ## Tokenize training data
> tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
> train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True,
>                                           return_offsets_mapping=True,
>                                           return_overflowing_tokens=True)
> 
> ## Find out start_position and end_position in encoded dataset
> def add_token_positions(encodings, answers):
>     start_positions = []
>     end_positions = []
> 
>     for idx in range(len(answers)):
>         start_positions.append(encodings.char_to_token(idx, answers[idx]['answer_start']))
>         if answers[idx]['answer_end'] is None:
>           end_positions.append(encodings.char_to_token(idx, answers[idx]['answer_end']))
>         else:
>           end_positions.append(encodings.char_to_token(idx, answers[idx]['answer_end'] - 1))
>         
>         #if None, the answer passage has been truncated due to words > 512 so setting last position as 511
>         if start_positions[-1] is None:
>             start_positions[-1] = tokenizer.model_max_length-1
>         if end_positions[-1] is None:
>             end_positions[-1] = tokenizer.model_max_length-1
>     
>     encodings.update({'start_positions': start_positions, 'end_positions': end_positions})
> 
> add_token_positions(train_encodings, train_answers)
> 
> ## Validate answers based on start_position and end_position with actual answer for some random index
> idx=8
> print(f'Actual context: {train_contexts[idx]}')
> print(f'Actual question: {train_questions[idx]}')
> print(f"Actual answer: {train_answers[idx]['text']}")
> 
> start_position=train_encodings.char_to_token(idx,train_answers[idx]['answer_start'])
> end_position =train_encodings.char_to_token(idx,train_answers[idx]['answer_end'])
> 
> ## ******This shows how start_position and end_position derived by using char_to_token() function is not correct******
> print(f"Answer after tokenization: {tokenizer.convert_ids_to_tokens(train_encodings['input_ids'][idx][start_position:end_position])}")
> ```

Since Pytorch removed SAVE_STATE_WARNING now it will pop up an error if install transformers==4.0.1. I use  transformers>=4.5 and it works