ValueError: char_to_token() is not available when using Python based tokenizers ; XLNetTokenizer and encodings.char_to_token bug ;
Author: akar5h
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- transformers version: 4.6.1
- Platform: Windows
- Python version:  3.8.8
- PyTorch version (GPU?):  1.8.1 , GPU enabled
- Tensorflow version (GPU?):  NA
- Using GPU in script?: Yes 
- Using distributed or parallel set-up in script?: No

### Who can help
 @patrickvonplaten 
 @LysandreJik 

## Information
Model I am using (Bert, XLNet ...): XLNet , "xlnet-base-cased"

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
my own modified script, but the issue can be reproduced as given below. 
encodings.char_to_token(i, answers[i]['answer_start'])
The error I get is : 
ValueError: char_to_token() is not available when using Python based tokenizers
- This issue is very similar to #9326 

The tasks I am working on is:
* [SQUAD ] an official GLUE/SQUaD task: (give the name)
*  A self-curated QA dataset in SQUaD format


Steps to reproduce the behavior:
Run the code snippet given below : 

```
import json
from pathlib import Path
from transformers import XLNetTokenizer, XLNetForQuestionAnsweringSimple
import torch

def read_squad(path):
    path = Path(path)
    with open(path, 'rb') as f:
        squad_dict = json.load(f)


    contexts = []
    questions = []
    answers = []
    for group in squad_dict['data']:
        for passage in group['paragraphs']:
            context = passage['context']
            for qa in passage['qas']:
                question = qa['question']
                for answer in qa['answers']:
                    contexts.append(context)
                    questions.append(question)
                    answers.append(answer)

    return contexts, questions, answers


train_contexts, train_questions, train_answers = read_squad('train-v2.0.json')
val_contexts, val_questions, val_answers = read_squad('dev-v2.0.json')


def add_end_idx(answers, contexts):
    for answer, context in zip(answers, contexts):
        gold_text = answer['text']
        start_idx = answer['answer_start']
        end_idx = start_idx + len(gold_text)

        # sometimes squad answers are off by a character or two – fix this
        if context[start_idx:end_idx] == gold_text:
            answer['answer_end'] = end_idx
        elif context[start_idx - 1:end_idx - 1] == gold_text:
            answer['answer_start'] = start_idx - 1
            answer['answer_end'] = end_idx - 1  # When the gold label is off by one character
        elif context[start_idx - 2:end_idx - 2] == gold_text:
            answer['answer_start'] = start_idx - 2
            answer['answer_end'] = end_idx - 2  # When the gold label is off by two characters


add_end_idx(train_answers, train_contexts)
add_end_idx(val_answers, val_contexts)

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

model_name = "xlnet-base-cased"
tokenizer = XLNetTokenizer.from_pretrained(model_name)
model = XLNetForQuestionAnsweringSimple.from_pretrained(model_name)
model.to(device)

train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True, max_length= 512)
val_encodings = tokenizer(val_contexts, val_questions, truncation=True, padding=True, max_length= 512)


def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))
        end_positions.append(encodings.char_to_token(i, answers[i]['answer_end'] - 1))

        # if None, the answer passage has been truncated
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})


add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)
```

![image](https://user-images.githubusercontent.com/19966604/122236270-ffd15280-cedb-11eb-8549-3b2856c03d4f.png)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
- encodings.char_to_token(i, answers[i]['answer_start']) should return some value 
- char_to_token should be not none in this case like in other tokenizers 

ValueError: char_to_token() is not available when using Python based tokenizers
encodings._encoding seems to be None


<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Why are you not using the fast tokenizer? The error message tells you that the feature `char_to_token` is not available for the slow (i.e. python) tokenizers because nobody has implemented it yet.

@cronoik , I ran the same with Fast Tokenizer  (XLNetTokenizerFast) on "xlnet-base-cased" , although char_to_token() was available this time , there seems to be some problem with XLNetTokenizerFast . 
``` 
   start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))
   end_positions.append(encodings.char_to_token(i, answers[i]['answer_end'] - 1))
```
While debugging this snippet from the above code,  I observed using **XLNetTokenizerFast on "xlnet-base-cased"**  that `encodings.char_to_token(i, answers[i]['answer_start'])`
is None for most of the cases . (90%) . The output is None ,  hence encoding["start_position"] and "end_position" have erronous values . 
and just changing the model, i.e, using **AutoTokenizer on "roberta-base"** , Unlike above I saw these valuse to be finite and not None . And I was further able to fine tune the model .  

Do you have some insight on this  ? 



There is to 95% nothing wrong with the tokenizer. You are just using it the wrong way. Please give us an example that leads to None. The `char_to_token` returns none when you ask for a whitespace position and you use a tokenizer that does not support whitespace.

Sure, Try these 2 code snippets , 

with XLNetTokenizerFast: 
```
import json
from pathlib import Path
from transformers import XLNetTokenizerFast, XLNetForQuestionAnsweringSimple
# from transformers import BigBirdTokenizerFast, BigBirdForQuestionAnswering

import torch

def read_squad(path):
    path = Path(path)
    with open(path, 'rb') as f:
        squad_dict = json.load(f)

    contexts = []
    questions = []
    answers = []
    for group in squad_dict['data']:
        for passage in group['paragraphs']:
            context = passage['context']
            for qa in passage['qas']:
                question = qa['question']
                for answer in qa['answers']:
                    contexts.append(context)
                    questions.append(question)
                    answers.append(answer)

    return contexts, questions, answers


train_contexts, train_questions, train_answers = read_squad('train-v2.0.json')
val_contexts, val_questions, val_answers = read_squad('dev-v2.0.json')


def add_end_idx(answers, contexts):
    for answer, context in zip(answers, contexts):
        gold_text = answer['text']
        start_idx = answer['answer_start']
        end_idx = start_idx + len(gold_text)

        # sometimes squad answers are off by a character or two – fix this
        if context[start_idx:end_idx] == gold_text:
            answer['answer_end'] = end_idx
        elif context[start_idx - 1:end_idx - 1] == gold_text:
            answer['answer_start'] = start_idx - 1
            answer['answer_end'] = end_idx - 1  # When the gold label is off by one character
        elif context[start_idx - 2:end_idx - 2] == gold_text:
            answer['answer_start'] = start_idx - 2
            answer['answer_end'] = end_idx - 2  # When the gold label is off by two characters


add_end_idx(train_answers, train_contexts)
add_end_idx(val_answers, val_contexts)

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

model_name = "xlnet-base-cased"
tokenizer = XLNetTokenizerFast.from_pretrained(model_name)
model = XLNetForQuestionAnsweringSimple.from_pretrained(model_name)
model.to(device)

train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True, max_length= 512)
val_encodings = tokenizer(val_contexts, val_questions, truncation=True, padding=True, max_length= 512)


def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))
        end_positions.append(encodings.char_to_token(i, answers[i]['answer_end'] - 1))

        # if None, the answer passage has been truncated
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})


add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)
```

With roberta-base:
```
import json
from pathlib import Path
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
# from transformers import BigBirdTokenizerFast, BigBirdForQuestionAnswering

import torch

def read_squad(path):
    path = Path(path)
    with open(path, 'rb') as f:
        squad_dict = json.load(f)

    contexts = []
    questions = []
    answers = []
    for group in squad_dict['data']:
        for passage in group['paragraphs']:
            context = passage['context']
            for qa in passage['qas']:
                question = qa['question']
                for answer in qa['answers']:
                    contexts.append(context)
                    questions.append(question)
                    answers.append(answer)

    return contexts, questions, answers


train_contexts, train_questions, train_answers = read_squad('train-v2.0.json')
val_contexts, val_questions, val_answers = read_squad('dev-v2.0.json')


def add_end_idx(answers, contexts):
    for answer, context in zip(answers, contexts):
        gold_text = answer['text']
        start_idx = answer['answer_start']
        end_idx = start_idx + len(gold_text)

        # sometimes squad answers are off by a character or two – fix this
        if context[start_idx:end_idx] == gold_text:
            answer['answer_end'] = end_idx
        elif context[start_idx - 1:end_idx - 1] == gold_text:
            answer['answer_start'] = start_idx - 1
            answer['answer_end'] = end_idx - 1  # When the gold label is off by one character
        elif context[start_idx - 2:end_idx - 2] == gold_text:
            answer['answer_start'] = start_idx - 2
            answer['answer_end'] = end_idx - 2  # When the gold label is off by two characters


add_end_idx(train_answers, train_contexts)
add_end_idx(val_answers, val_contexts)

device = torch.device('cpu')

model_name = "roberta-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
model.to(device)

train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True, max_length= 512)
val_encodings = tokenizer(val_contexts, val_questions, truncation=True, padding=True, max_length= 512)


def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))
        end_positions.append(encodings.char_to_token(i, answers[i]['answer_end'] - 1))

        # if None, the answer passage has been truncated
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})


add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)
```

In the two snippets above, check the value of  "start_positions" ,  "end_positions"  variables  in "add_token_positions" function  after its final iteration , and compare them .  Its tokenizer.model_max_length for most cases in XLNet one. 
Now why is it that in tokenizer (specifically , encodings.char_to_token(i, answers[i]['answer_start']))  its returning finite values in roberta  , and with other tokenizer its None .  All that was changed was Tokenizer    
[ encodings.char_to_token(i, answers[i]['answer_start']) is None for xlnet model and , not for roberta . 



Please give us an example of your text that produces None. You have already shown us your code.


[train-v2.0.txt](https://github.com/huggingface/transformers/files/6671676/train-v2.0.txt)
Consider this slice of Squad2.0 dataset , roughly 65 contexts and their qas .  (change file from .txt to json ) 
I'm working on the complete Squad2.0 dataset  , but this json will reproduce the issue.

much help 

Can someone help me understand the purpose "add_token_position" function?  I've read multiple articles and watched videos and they all mention "we need to add the token position" but I honestly don't understand that explanation.  For example, if we try and fine-tune a bert-base-uncased the start_position for train_context[0] is 67 and the end_position is 70 (subtracting -1 to account for space).  I'm fairly certain these numbers represent indices but indices of what and in what list?  Thanks for your help. 

Any update @cronoik on why XLnet tokenizer is returning None because it still is returning the same.

Sorry, I found the reaction of @akar5h  very unfriendly and decided to ignore this issue I'll look into it later.