Exposing prepare_for_model for both slow & fast tokenizers
Author: LysandreJik
Pull request: yes
Labels: name: Core: Tokenization

With version v3.0.0, two breaking changes that could have been avoided have been introduced. After discussion with @n1to and @thomwolf, this PR aims to revert these changes, by implementing two changes:

- The `prepare_for_model` method for both slow and tokenizers is now publicly exposed (it was only the case for the slow tokenizers before v3.0.0)
- The truncation methods now default to `longest_first` instead of `first_only` .

This PR adds two tests, for both Python and Rust tokenizers:

- Assert that `tokenizer.prepare_for_model(tokenizer.encode(x)) == tokenizer.encode_plus(x)`
- Assert that the output of `prepare_for_model` for rust and python tokenizers is equal.

closes https://github.com/huggingface/transformers/issues/5377
closes https://github.com/huggingface/transformers/issues/5447
closes https://github.com/huggingface/transformers/issues/5460

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5479?src=pr&el=h1) Report
> Merging [#5479](https://codecov.io/gh/huggingface/transformers/pull/5479?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ef0e9d806c51059b07b98cb0279a20d3ba3cbc1d&el=desc) will **increase** coverage by `0.43%`.
> The diff coverage is `89.53%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5479/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5479?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5479      +/-   ##
==========================================
+ Coverage   77.36%   77.80%   +0.43%     
==========================================
  Files         141      141              
  Lines       24617    24632      +15     
==========================================
+ Hits        19045    19164     +119     
+ Misses       5572     5468     -104     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5479?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `93.60% <89.28%> (-0.62%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `91.16% <100.00%> (+0.94%)` | :arrow_up: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `28.84% <0.00%> (-58.66%)` | :arrow_down: |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.40% <0.00%> (+0.71%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.43% <0.00%> (+1.25%)` | :arrow_up: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `97.77% <0.00%> (+2.22%)` | :arrow_up: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `92.23% <0.00%> (+2.28%)` | :arrow_up: |
| [src/transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcm9iZXJ0YS5weQ==) | `98.63% <0.00%> (+4.10%)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `55.79% <0.00%> (+27.58%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/huggingface/transformers/pull/5479/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5479?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5479?src=pr&el=footer). Last update [ef0e9d8...ecdc965](https://codecov.io/gh/huggingface/transformers/pull/5479?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).