HF's Flan-T5 implementation doesn't support Chinese or code despite being trained on it
Author: michaelroyzen
Pull request: no

### System Info

transformers == 4.26.1
pytorch == 1.13.1

### Who can help?

@ArthurZucker @younesbelkada 



### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-xxl")
tokenizer.decode(tokenizer("你好你好吗").input_ids)
```

returns `<unk>
`.

Similarly, the tokenizer can't encode curly braces (`{` or `}`) or `\n` or `\t`, making it useless for code. Is the tokenizer included with the model the right one?

### Expected behavior

The tokenizer should be able to encode Asian languages (including Chinese) as well as code. The model was trained on both according to the paper. Did you port the proper tokenizer from the T5x repo?

I would appreciate your help.

Comments:
Hey! Thanks for posting. The original tokenizer does not support chinese (it only supports 4 language I think) either. 
Here is a minimal reproducing script using the vocabulary path provided in the `t5_1_1_base.gin` that is used for all of the Flan T5 (according to github). 
```python 
>>> import seqio
>>> vocabulary = seqio.SentencePieceVocabulary("gs://t5-data/vocabs/cc_all.32000.100extra/sentencepiece.model")
>>> vocabulary.tokenizer.encode("你好你好吗")
[3, 2]
>>> vocabulary.tokenizer.decode(vocabulary.tokenizer.encode("你好你好吗"))
' ⁇ '
```
We probably made a mistake in the `tags` of the model that should not include these. The paper does not mention anything else, and I tested with the mT5 tokenizer without avail. 
Will try too look a bit more into this. 

They probably did not release the multilingual finetune checkpoints : we only have token vocabulary of 32 000 instead of 250 000 used for mT5 their multilingual tokenizers.

Thanks for the clarification @ArthurZucker. But the paper did mention the model being fine-tuned on code -- but I don't see how that is possible if the model can't support newlines, brackets, or tabs as tokens.

Yes! The paper seems to mention multilingual and code, but I dug and could not reproduce anything... Again a minimal reproduction script: 
```python 
>>> import seqio
>>> vocabulary = seqio.SentencePieceVocabulary("gs://t5-data/vocabs/cc_all.32000.100extra/sentencepiece.model")
>>> vocabulary.tokenizer.encode("if True:\n\tprint('Wow')")
[3, 99, 10998, 10, 2281, 599, 31, 518, 2381, 31, 61]
>>> vocabulary.tokenizer.decode(vocabulary.tokenizer.encode("if True:\n\tprint('Wow')"))
"if True: print('Wow')"
```
I might not be using some special arguments but I am not familiar with the black box seqio 😅 

Right -- this is exactly what I'm talking about. Is there any way to reach out to the authors and figure this out? Seems pretty important that core functionality isn't working. And in the meantime, perhaps the model card should remove mentions to multilingual capabilities that it can't actually support.

We merged a lot of PRs on the hub to fix this. Marking as resolved! Thanks for reporting

@ArthurZucker 
hi, I have the same problem with flan-t5 to segment Chinese. 
is there some thing I should do to resolve this? like upgrade pip package.
or  other things?

You should just read this issue 😉 nothing we can do about it unfortunately 

@ArthurZucker ok, I read that commit,and I get it, thank you

I'm using the latest version as much as I can, but I'm still having this issue. Is there anything else I can do next?

It is not supported.

Oh sorry, I am reading the code for a paper that uses t5. It turns out that they are using mt5, but the model is still referred to as t5. I overlooked this point and apologize for my carelessness.

No worries 🤗