Auto tokenizer for question-context tokenization
Author: Arij-Aladel
Pull request: no

@patrickvonplaten, @patil-suraj, @LysandreJik
Hello!
I have trained this [tokenizer](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer-2)

I have a question answering task using T5 and I need the question and context to be tokenized as T5Tokenizer do. I mean `quesion_ids
context_ids
<pad>`  I did the following 



```
tokenizer = AutoTokenizer.from_pretrained(path_to_tokenizer_json_file, config=AutoConfig.from_pretrained(path_to_model_config))
query_tok = 'Do you go to school?'
doc_tok = 'What could be a second sequence to the uninteresting text, What could \
           be a second sequence to the uninteresting text'

query_tok = " ".join(query_tok.split()[:min(5,len(query_tok.split()))])
doc_tok = " ".join(doc_tok.split()[:min(9,len(doc_tok.split()))])
print("query_tok: ", query_tok)
print("doc_tok: ", doc_tok)

ids = tokenizer(
      query_tok+'
',
      doc_tok+'
',
      max_length=15,
      padding='max_length',
      truncation="only_second",   #   need to experiement
      return_attention_mask=True,
      add_special_tokens=True,
      return_tensors="pt"
      )
print("ids: ",ids)
print("ids shape: ",ids['input_ids'].shape)

print("Len ids: ",len(ids['input_ids'].flatten()), len(ids['input_ids'].flatten()) - 9)

print(tokenizer.decode([id for id in ids['input_ids'].flatten()]))
```


the result was 

> query_tok:  Do you go to school?
> doc_tok:  What could be a second sequence to the uninteresting
> ids:  {'input_ids': tensor([[   3,   11,   36,  316,  475,   16,  156,  187,    1,  219,  312,   63,
>             3,    8,  114, 1906,   16,    4,  352, 6347,  483,   38,    0,    0,
>             0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,
>          0]])}
> ids shape:  torch.Size([1, 25])
> Len ids:  25 16
> do you go to school?
 what could be a second sequence to the uninteresting<pad><pad><pad>
> 

How to make the tokenizer add 
 to the end of the context before padding or after truncation?


Comments:
Hey @Arij-Aladel, 

Sorry I'm having trouble to fully understand your question here. Could you clarify a bit what output you would expect? E.g. in your opinion what should:

```bash
print(tokenizer.decode([id for id in ids['input_ids'].flatten()]))
```

print out?

@patrickvonplaten Thanks for your response. Sorry for not clear explanation. I hope this notebook will explain better than words the differences between T5tokenizer and autotokenizer. https://colab.research.google.com/drive/1Gro30z9tbyUUxXR4-bGtKhJzwnEVsXR4?usp=sharing

Hey @Arij-Aladel,

Could you please try to give a minimal reproducible code snippet that clearly shows where the problem is? So far I've understood that there is a problem with tokenizer (probably T5) in the context of question answering, but I don't know exactly what the problem is.

The notebook starts by importing `pytorch-lightning` which is not related to this issue, which makes me stop looking into the google colab.

Please keep in mind that we are getting 100s of pings on GitHub each day and we need to optimize our time when solving issues. It would be very nice if you could try to have a minimal reproducible code snippet here. 

It's always nice to add a code example just like you did above, but then please make sure it is as easy to understand as possible and that everybody can copy-paste and run it.
E.g. copying your issue above and adding comments to each line which should be improved so that we can better help you

```py
# I cannot use `path_to_tokenizer_json_file` -> please give me something that I can run locally
tokenizer = AutoTokenizer.from_pretrained("t5-small")

query_tok = 'Do you go to school?'
doc_tok = 'What could be a second sequence to the uninteresting text, What could \
           be a second sequence to the uninteresting text'

# Why do you do this `min(.....)` stuff here??? Please remove this if it's not relevant to the issue
# or explain why. This is definitely not a "normal" case
query_tok = " ".join(query_tok.split()[:min(5,len(query_tok.split()))])
doc_tok = " ".join(doc_tok.split()[:min(9,len(doc_tok.split()))])

print("query_tok: ", query_tok)
print("doc_tok: ", doc_tok)

# Why do you append '
` here?
ids = tokenizer(
      query_tok+'
',
      doc_tok+'
',
      max_length=15,
      padding='max_length',
      truncation="only_second",   #   need to experiement
      return_attention_mask=True,
      add_special_tokens=True,
      return_tensors="pt"
      )
print("ids: ",ids)
print("ids shape: ",ids['input_ids'].shape)

# Why do I need to know the length - why `-9`?
print("Len ids: ",len(ids['input_ids'].flatten()), len(ids['input_ids'].flatten()) - 9)

# Why do you do `[id in id in ids["..."]]` instead of `batch_decode(...)`?
print(tokenizer.decode([id for id in ids['input_ids'].flatten()]))
```

@patrickvonplaten   https://drive.google.com/drive/folders/1bQVwilQ3RAah7kvnzLSKbdx92s6bv0wX?usp=sharing

Is my question clear now?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.