Where is the documentation on migrating to the 3.0 tokenizer API?
Author: githubrandomuser2017
Pull request: no

I see that you folks have completely changed the API to do tokenizing, e.g. for BertTokenizer. I have a lot of code using the two methods `encode_plus()` and `batch_encode_plus()`, and when I went to the [documentation](https://huggingface.co/transformers/main_classes/tokenizer.html) to look up an argument, I found that these methods are completely gone. All that remains is a little blurb saying:

> `BatchEncoding` holds the output of the tokenizer’s encoding methods (`__call__`, `encode_plus` and `batch_encode_plus`) and is derived from a Python dictionary.

Are these two methods deprecated now? Did you post a migration guide for users? 

On the main [Huggingface Transformers page](https://github.com/huggingface/transformers), you have sections for `Migrating from pytorch-transformers to transformers` and `Migrating from pytorch-pretrained-bert to transformers`, so it's not like there's no precedent for you to provide some information to users on major API changes.

Comments:
Hi @githubrandomuser2017 , AFAIK these methods will still work in v.3.0 as backward compatibility is maintained. This can help
https://huggingface.co/transformers/preprocessing.html

@patil-suraj The page you mentioned (https://huggingface.co/transformers/preprocessing.html) doesn't mention anything about the missing functions (`encode_plus()` and `batch_encode_plus()`). Also, the functions are not in the tokenizer documentation page anymore (https://huggingface.co/transformers/main_classes/tokenizer.html). I need to look up some arguments.

for me `encode_plus()` and `batch_encode_plus()` are working as expected in v3.

Also as far as I understand 
`tokenizer` object is now a callable and by default it behaves as encode_plus, i.e it returns `input_ids` along with `attention_mask`, `token_type_ids` etc, this can be controlled using `return_attention_mask`, `return_token_type_ids` arguments.

So by default if you provide a single example to `tokenizer` it will behave as encode_plus and if you provide a batch of examples it'll behave like `batch_encode_plus`. 

padding and truncation is now controlled using `padding` and `truncation` arguments.

So 
```python3
tokenizer.encode_plus("some text", max_length=512, pad_to_max_length=True)
``` 
is now equivalent to 
```python3
tokenizer("some text", max_length=512, padding="max_length", truncation=True)
````

and
```python3
tokenizer.batch_encode_plus(["some text", "some other text"], max_length=512, pad_to_max_length=True)
``` 
is equivalent to 
```python3
tokenizer(["some text", "some other text"],  max_length=512, padding="max_length", truncation=True)
````

https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.__call__

Indeed, @patil-suraj is right, `__call__` is a wrapper on top of both `encode_plus` and `batch_encode_plus` which will dispatch depending on whether you are providing a single example or a batch.

We don't promote the use of `encode_plus` and `batch_encode_plus` but we kept backward compatibility for them so they are still available and behave identically as in transformers version `2.X`.

If you want to access the doc for them, use the doc for version `2.X`: https://huggingface.co/transformers/v2.11.0/main_classes/tokenizer.html?highlight=batch_encode_plus#transformers.PreTrainedTokenizer.batch_encode_plus

But we are right we should add a migration guide, in particular for the padding and truncation commands. We can use the detailed migration guide in the description of this PR as a basis https://github.com/huggingface/transformers/pull/4510.

@sgugger do you want to give it a look when you have some bandwidth?


Will try do something before the end of the week.

I took a stab at it and added it to our brand new forum, look [here](https://discuss.huggingface.co/t/migration-guide-from-v2-x-to-v3-x-for-the-tokenizer-api/55).

Wonderful! Thank you. 

For the forum, do I use my Github credentials?

No, you need to create an account on https://huggingface.co/ if you don't have one already (same as for uploading models to the hub).

Thank you. Can you please put that fact (creating a huggingface.co account) on the main forum webpage? https://discuss.huggingface.co/