T5Tokenizer: Using cls_token, but it is not set yet.
Author: BramVanroy
Pull request: no
Labels: name: wontfix

## 🐛 Bug

Model I am using (Bert, XLNet....): T5

## To Reproduce

Steps to reproduce the behavior:

1. Load T5Tokenizer
2. Try getting the CLS or SEP token: `tokenizer.sep_token` or `tokenizer.cls_token`
3. An error will be raised "Using cls_token, but it is not set yet."

Running the latest commit on the master branch.

I imagine that the T5 implementation is not complete yet, but I thought I'd point it out anyway. I figured that T5 expects the same input as BERT, as stated in the source:

https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/modeling_t5.py#L651-L658


As an aside, I saw this incorrect mention of XLNetTokenizer in the T5Tokenizer. Probably overlooked while working on adding _yet another model_ to transformers. You guys are crazy!

https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/tokenization_t5.py#L112-L120


Comments:
I'm **not** able to import T5Tokenizer in my environment: I received an **ImportError** exception.
I'm using Python 3.6.9, OS Ubuntu 16.04, Transformers 2.2.2 (installed now with `pip install transformers`), PyTorch 1.3.1 and TensorFlow 2.0. What am I missing?
The stack trace is the following:
```
>>> import transformers
>>> transformers.__version__
>>> '2.2.2'
>>> from transformers import T5Tokenizer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'T5Tokenizer'
```

I can't import `T5Model` as well.

> ##  Bug
> Model I am using (Bert, XLNet....): T5
> 
> ## To Reproduce
> Steps to reproduce the behavior:
> 
> 1. Load T5Tokenizer
> 2. Try getting the CLS or SEP token: `tokenizer.sep_token` or `tokenizer.cls_token`
> 3. An error will be raised "Using cls_token, but it is not set yet."
> 
> Running the latest commit on the master branch.
> 
> I imagine that the T5 implementation is not complete yet, but I thought I'd point it out anyway. I figured that T5 expects the same input as BERT, as stated in the source:
> 
> https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/modeling_t5.py#L651-L658
> 
> As an aside, I saw this incorrect mention of XLNetTokenizer in the T5Tokenizer. Probably overlooked while working on adding _yet another model_ to transformers. You guys are crazy!
> 
> https://github.com/huggingface/transformers/blob/e92bcb7eb6c5b9b6ed313cc74abaab50b3dc674f/transformers/tokenization_t5.py#L112-L120

@TheEdoardo93 T5 is _not_ part of the 2.2.2 release. In other words, it is not part of a PyPi release yet. You'll need to uninstall transformers and install from this repo.

I've installed Transformers from source with `pip install git+https://github.com/huggingface/transformers.git` and now it works as expected! Thank you

> @TheEdoardo93 T5 is _not_ part of the 2.2.2 release. In other words, it is not part of a PyPi release yet. You'll need to uninstall transformers and install from this repo.

Yes, thanks for trying it already @BramVanroy.

Currently, T5 is still missing a few features to be easily usable.
I'll add them over the coming days/weeks.

Here is what we still need and we plan to add:
- a clean sequence generation API (T5 is designed to be used in a text generation setting, you should read the paper if you haven't already by the way, it's a great paper) <= working on this right now at #1840 
- a clean way to do model parallelism to spread the model on several GPUs. For instance, the biggest checkpoint is 42 GB (!) so you'll need a few GPU only to load the model <= Working on this after the seq generation (first draft at #2165 )
- a script to pre-process GLUE/SQUAD to set them in text generation setting <= Later

I just skimmed through https://github.com/huggingface/transformers/pull/1840 and it's great to see how you're focusing on the user-experience  @thomwolf. I think that that's a very important thing, together with well structured and written documentation. The field is changing rapidly, and especially for people who are working more on the linguistics side of things or who are just getting into transformers, a clear API is a must to get started. Thanks a lot for that (and your and the team's work in general, of course)! 

Concerning https://github.com/huggingface/transformers/pull/2165, why don't you just require people to rent TPUv3-2048 with 32**TB** of memory? No biggey. I kid of course! Nice to see that efforts are underway to bring these big models to consumers as well. This work completely goes over my head, but I am curious to see what the end result is. Would that mean that we could load that 42GB checkpoint on our 4x V100 (16GB each)?

Perhaps this issue should stay open and be closed when T5 is finished?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.