feat&fix(tokenization): add new consistent API for encoding and decoding related methods.
Author: scruel
Pull request: yes

Add new methods `consistent_encode`/`consistenct_decode` and `consistent_encode_batch`/`consistent_decode_batch` to make the public API consistent.

Notice#1: To prevent to create a PR bomb, I decide to create this PR as a "new feature", consider that we have many code and tests which are using tokenization related methods, so the backward compatibility has not been considered in it (we may can have it in v5), including consider letting `encode` support both single and batches. Don't forget to check the [comment](https://github.com/huggingface/transformers/issues/28635#issuecomment-1910021077) that I post in the related issue!
Notice#2: This PR is still incomplete, I think I will need some feedback before I can continue, so you can see `TODO`s in it and the documentation is not updated yet.

Fixes #28635


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @younesbelkada



Comments:
> There are too many changes for a single PR

Agree, it changed a lot. If you don't like the changes I made, I will just revert them back to the original.

> we can test if the input is a BatchEncoding or test the keys of the dictionary passed to the decode function

Why the input will be a `BatchEncoding` or `dict` to `decode` func? Consider `encode` method will only return a list of integers.


> if the input is a list of strings just call batch_decode as we know for a fact that decode won't work for slow

I don't understand, what is the meaning of "`decode` won't work for `slow`"? If you were mentioning the differences of `slow/fast` methods, I think we shall consider them both even one of them is not supported, so we just can't handle it welly within one method, consider a list of strings will have different meanings according to how we treat it (`list[TextInput]` or `PreTokenizedInput`).


Oh, I think you misunderstood this PR, could you please read the [comment](https://github.com/huggingface/transformers/issues/28635#issuecomment-1910021077) again, and may check [test code](https://github.com/huggingface/transformers/blob/bff35e5ea3c2856f14ddf6926568098cae8b2678/tests/test_tokenization_common.py#L987-L1065) to know what I want to achieve, thanks.
In short, it's not to combine both encode/encode_plus/encode_batch related methods, it's just to make the API more consistent.

Ah yes then I misunderstood. 
It's not that I don't like the changes but more that I am not sure offering more freedom is where we want to go! I'd rather restrict what inputs / type of inputs can be taken. 
But if you want to work on combine both `decode/decode_batch` that would be more impactful I think! 🤗 

> It's not that I don't like the changes but more that I am not sure offering more freedom is where we want to go! I'd rather restrict what inputs / type of inputs can be taken.

On the contrary, this PR will limit user's freedom and already restricted the inputs.

> But if you want to work on combine both decode/decode_batch that would be more impactful I think! 🤗

Aha, I do want to combine the both when I started this, however I think we can't (or we shouldn't) do this, as I mentioned above for the special input -- a list of strings.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ArthurZucker Any update?

Sorry not yet. I'll come back to this but again, not really high priority at the moment!

> Sorry not yet. I'll come back to this but again, not really high priority at the moment!

Ok, whenever you want to start, you can just take code from this PR.