Enable option for subword regularization in more tokenizers.
Author: PhilipMay
Pull request: yes

see https://github.com/huggingface/transformers/pull/11149#pullrequestreview-643686428

## To-do
### `AlbertTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] remove obscure function argument called `sample`
- [x] check
- [x] refactor test to follow DRY
### `BarthezTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `BertGenerationTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] remove obscure function argument called `sample`
- [x] check
- [x] refactor test to follow DRY
### `BigBirdTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] remove obscure function argument called `sample`
- [x] check
- [x] refactor test to follow DRY
### `CamembertTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `DebertaV2Tokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `M2M100Tokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `MarianTokenizer` - has src and target tokenizer
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `MBart50Tokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `PegasusTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] remove obscure function argument called `sample`
- [x] check
- [x] refactor test to follow DRY
### `ReformerTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] remove obscure function argument called `sample`
- [x] check
- [x] refactor test to follow DRY
### `Speech2TextTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `T5Tokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] remove obscure function argument called `sample`
- [x] check
- [x] refactor test to follow DRY
### `XLMProphetNetTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] check
- [x] refactor test to follow DRY
- 
remove obscure function argument called `sample`

### `XLNetTokenizer`
- [x] add `sp_model_kwargs` param with test
- [x] add pickle support with test
- [x] remove obscure function argument called `sample`
- [x] check
- [x] refactor test to follow DRY
### `XML RoBERTa`
- [x] refactor test to follow DRY
### General
- [x] check if we changed all tokenizers
- [x] add typing
- [x] check if tok. is used in other functions
- [x] also add changes to XLM RoBERTa tokenizer
### After review
- [x] fix type comments with default `None`
- [x] possibly remove `test_sentencepiece_skip_back_convert_check`

Comments:
I found this somehow obscure function argument called `sample` at `AlbertTokenizer`:

https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/albert/tokenization_albert.py#L189

It seems to enable subword regularization but with fixed parameters for `nbest_size` and `alpha`.

https://github.com/google/sentencepiece/blob/351600c2971401f4e849147579aa1b5d42f614e1/python/src/sentencepiece/__init__.py#L110-L111

I would remove that `sample` parameter and replace that with my solution which is more flexible. But that would mean we have a breaking change. As an alternative I could add my solution but keep the `sample` argument. But that would add more complexity to the code.

What do you think? @sgugger @LysandreJik @stefan-it 

PS: Same here:

https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/bert_generation/tokenization_bert_generation.py#L113

https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/big_bird/tokenization_big_bird.py#L143

https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/pegasus/tokenization_pegasus.py#L169

https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/reformer/tokenization_reformer.py#L109

https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/t5/tokenization_t5.py#L237

https://github.com/huggingface/transformers/blob/52166f672ed337934d90cc525c226d93209e0923/src/transformers/models/xlnet/tokenization_xlnet.py#L191

This argument is not called from anywhere so it's only accessible if users somehow rewrote the tokenize method to pass it along to the private method `_tokenize`. Therefore I think it's fine to do the breaking change and clean up the code using `sample=True`, but let's see what @patrickvonplaten and @LysandreJik think before going forward (note that Lysandre is on vacation until this Wednesday so he'll reply at the end of the week :-) ).

Yes, removing the `sample` and cleaning up the `_tokenize()` method sounds good to me. As @sgugger said, it is private and nowhere is a `sample` or a `**kwargs` passed to that method.

> Yes, removing the `sample` and cleaning up the `_tokenize()` method sounds good to me. As @sgugger said, it is private and nowhere is a `sample` or a `**kwargs` passed to that method.

Agree!

rebase upstream/master done

> Yes, LGTM! Thanks a lot.

Hey @LysandreJik - this is not done yet. Please do not merge now. ;-)

Oh, I was misled! There are indeed a few tokenizers remaining. Thank you for letting me know!

This is ready to be merged from my point of view.

Can you take care of the merge conflicts? Will review tomorrow :-)

> Can you take care of the merge conflicts? Will review tomorrow :-)

@sgugger All conflicts resolved & green CI




> Great work on the tests, this is great. The tests could indeed be refactored in a common test if you feel like it.

I will refactor the tests the next days. Shame on me that I criticized the lack of DRY in the tokenizers but did not follow the DRY principle in the tests.

This is strange:

`FAILED tests/test_hf_api.py::HfApiEndpointsTest::test_list_repos_objs - reque...`

See here: https://app.circleci.com/pipelines/github/huggingface/transformers/23276/workflows/bf1ad505-efdc-4394-8852-a07702b9f5be/jobs/209965/parallel-runs/0/steps/0-108

Will trigget CI again,,,

@LysandreJik @sgugger Tests are refactored and DRY now. CI is green again.
IMO ready for merge.

Maybe you want to investigate the flaky test (see my comment above).