add `**kwargs` argument in some functions in `tokenization_utils.py`
Author: yusuke1997
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Despite `**kwargs` being set in the `prepare_for_model` function in `tokenization_utils_base.py`, there was an issue where `**kwargs` were not being set in some functions within `tokenization_utils.py`, resulting in `**kwargs` not being propagated to the `prepare_for_model` function.

This PR eliminates the need to copy multiple functions to set `**kwargs` when customizing the `prepare_for_model function, we can make a more concise code.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker
@sgugger


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22672). All of your documentation changes will be reflected on that endpoint.

Can you elaborate and give us a sample of code that fails before this PR? Thanks.

Thanks for the reply! 

In my case, I need to consider adding custom special tags within the `def build_inputs_with_special_tokens(...)`, for example, whether to add `<Language ID>` in addition to `<eos>` and `<bos>` tags. As I need to switch frequently, I decided to control this by an argument like the following code.
```python
    def build_inputs_with_special_tokens(
            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, prepend: bool = True
    ) -> List[int]:
```
```python
tokenizer('Hi world.', add_special_tokens=True, prepend=False)
# {'input_ids': [4218, 1381, 6, 2], 'attention_mask': [1, 1, 1, 1]}
tokenizer('Hi world.', add_special_tokens=True, prepend=True)
# {'input_ids': [806, 4218, 1381, 6, 2], 'attention_mask': [1, 1, 1, 1, 1]}
```
The reason why the `def prepare_for_tokenization(...)` function did not handle this is that these tags need to be treated as special_tokens, just like `<eos>` and `<bos>` tags.

`def build_inputs_with_special_tokens(...)` is executed by `def prepare_for_model(...)`, so I rewrite that function slightly.
```python
def prepare_for_model(
        ...
        prepend: bool = True, #add this line
        **kwargs,
    ) -> BatchEncoding:

        ...

        # Add special tokens
        if add_special_tokens:
            sequence = self.build_inputs_with_special_tokens(ids, pair_ids, prepend = prepend)
            token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids, prepend = prepend)

        ...
```
I would like it to work with this modification only, but the functions using `def prepare_for_model(...)` ( `def _encode_plus(...)` and `def _batch_prepare_for_model(...)` and `def _batch_encode_plus(...)` in `tokenisation_utils.py`), `** kwargs` are not passed on, which is causing it not to work.

The following code is an example of this issue in `def _encode_plus(...)`
```python
    def _encode_plus(
        self,
        text: Union[TextInput, PreTokenizedInput, EncodedInput],
        text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None,
        add_special_tokens: bool = True,
        padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
        truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE,
        max_length: Optional[int] = None,
        stride: int = 0,
        is_split_into_words: bool = False,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        return_token_type_ids: Optional[bool] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_offsets_mapping: bool = False,
        return_length: bool = False,
        verbose: bool = True,
        **kwargs,
    ) -> BatchEncoding:

        ...

        return self.prepare_for_model(
            first_ids,
            pair_ids=second_ids,
            add_special_tokens=add_special_tokens,
            padding=padding_strategy.value,
            truncation=truncation_strategy.value,
            max_length=max_length,
            stride=stride,
            pad_to_multiple_of=pad_to_multiple_of,
            return_tensors=return_tensors,
            prepend_batch_axis=True,
            return_attention_mask=return_attention_mask,
            return_token_type_ids=return_token_type_ids,
            return_overflowing_tokens=return_overflowing_tokens,
            return_special_tokens_mask=return_special_tokens_mask,
            return_length=return_length,
            verbose=verbose, 
            **kwargs, # ADD THIS LINE!!!!!
        )

```
Therefore, by adding this PR fix, `**kwargs` will be passed to `def prepare_for_model(...)` and solve this problem.

In addition, although `**kwargs` are set to `def prepare_for_model(...)` in the code, there was no mechanism in place to use `**kwargs` simply, which is resolved in this PR.


Overall, this PR resolves the issue of `**kwargs` not being propagated in some functions within `tokenization_utils.py`, providing a more efficient and streamlined way to customize the `def prepare_for_model(...)` function.

Thank you for reviewing this PR!!!!

I'm a bit wary about passing all those kwargs since there could be unrelated/mispelled arguments that the user wouldn't get an error about then. The easiest is probably for you copy paste those methods and do the change in your subclass of the tokenizer, since, if I understand correctly, you are writing your own subclass of the tokenizer with an overloaded method.

Thank you for your comments!
I explain that there is no need to worry about this PR.
> I'm a bit wary about passing all those kwargs since there could be unrelated/mispelled arguments that the user wouldn't get an error about then.

As all the functions fixed in this PR already have the `**kwargs` argument, the worry is a problem that could well occur in existing systems. This is not a new problem caused by this PR.
**It is not an attempt to set new `**kwargs` in `def prepare_for_model(...)`, but a PR to leverage the `**kwargs` already set in `def prepare_for_model(...)`.**

>  The easiest is probably for you copy paste those methods and do the change in your subclass of the tokenizer

If only a few functions with few codes need to be edited, it would certainly be the easiest way. 
However, in this case,  to utilize the `**kwargs` in `def prepare_for_model(...)`, it would require copy-pasting hundreds of lines of code with minimal modifications. From code maintenance, it is important to avoid such duplications as much as possible.

The changes made by this PR are not breaking changes, so they will work on all existing models. Additionally, the worry you mentioned is not specific to this PR, as it is a potential issue that could also occur in existing systems.

For the above reasons, respecting your worry, but I still believe that the benefits of utilizing `**kwargs` in `def prepare_for_model(...)` outweigh the risks.

I hope that this PR will contribute to the continued improvement and maintenance of the system. 
Thank you for your comments and looking forward to your positive feedback.

cc @LysandreJik if you have a different opinion.

Hey @yusuke1997, thank you for your PR!

I would err on the side of caution here with having `**kwargs` on every method. When we add `"**kwargs` to a method, we're forfeiting argument validation in favor of convenience, which is likely to bite us later. 

A very simple example of a bug this PR would introduce can be seen with the following script, where I misspelled `return_token_type_ids`:

```python
In [1]: from transformers import GPT2TokenizerFast

In [2]: tok = GPT2TokenizerFast.from_pretrained('gpt2')

In [3]: tok("hey", return_token_type_id=False)
```

On `main`, this errors with `TypeError: PreTrainedTokenizerFast._batch_encode_plus() got an unexpected keyword argument 'return_token_type_id'`.

On this PR, this doesn't error and returns 
```
Out[3]: {'input_ids': [20342], 'attention_mask': [1]}
```
which is false.


---

Your `prepend` approach can still work, but we would strongly recommend that you do it on a per-architecture approach than to try and patch a global method.

If you're using GPT2 for example, then I would advise overriding the `build_inputs_with_special_tokens` within this tokenizer directly.

Thank you @LysandreJik for a very helpful review!!

I checked. Certainly, on `main` `PreTrainedTokenizerFast` has a mechanism to raise an error when input misspelled arguments.
This PR eliminates such errors and it is able to handle any type of input.
I would like to reiterate that the purpose of this PR was to utilize `def prepare_for_model(...)`, and upon further review of the code, I found that `def prepare_for_model(...)` is not being utilized within `PreTrainedTokenizerFast`.
The modification made for `PreTrainedTokenizerFast` was superfluous. I retract the changes.

However, I still believe that the changes made to `tokenization_utils.py` are meaningful, non-breaking changes, and safe.
This is because even in the `main` `PretrainedTokenizer`, misspelled or invalid arguments will pass with only a warning raised.
```python
from transformers import GPT2Tokenizer
tok = GPT2Tokenizer.from_pretrained('gpt2')
print(tok("hey", return_token_type_id=False))
# Keyword arguments {'return_token_type_id': False} not recognized.
# {'input_ids': [20342], 'attention_mask': [1]}
```
**This warning is no different in `main` and in this PR.**
To avoid this warning, we need to pop the desired argument to use in `def prepare_for_tokenization(...)`.
This part is also no different in the current `main` and after PR.

**The purpose of this PR is to utilize the `**kwargs` in `def prepare_for_model(...)` function.**
If it works by simply overriding `def build_inputs_with_special_tokens(...)`, it couldn't be easier than that.
However, currently, just adding a new argument to `def build_inputs_with_special_tokens(...)` requires a number of function changes.

This restriction of not being able to utilize the `def prepare_for_model(...)` arguments apply not only to this function but also to all the functions executed by `def prepare_for_model(...)`, I think it is causing potential drawbacks.

**Furthermore, I confirmed that there are no destructive elements, such as error displays or other potentially harmful changes, involved in implementing such a mechanism. And any input to the already set `**kwargs` in `def prepare_for_model(...)` will not have any adverse effects.**
So, I believe it is reasonable to establish a mechanism for passing `**kwargs` in order to utilize `def prepare_for_model(...)` (and functions executed by it). 
Therefore, I think there is no need to fear the part that you are cautious about.

If any concerns have been resolved, I would sincerely hope you will think about again it.
Thank you very much for your attention and time!!






In addition,
Your point about the error not being raised is actually due to the different handling of unrelated/misspelled arguments between `PretrainedTokenizer` and `PretrainedTokenizerFast`.
I am of the opinion that **if `PretrainedTokenizer` is set up to generate warnings instead of errors, then `PretrainedTokenizerFast` should also generate warnings instead of errors**, and vice versa.
In other words, the behavior of both is different, and this is what the PR has revealed.

What do you think about this point? What would be the best solution?
For the purpose of furthering my understanding of this matter, would you kindly explain it to me? I am genuinely curious.

Sorry (miss-clicked the tag)! A small part of my answer is that this looks good to me, but at the same time I think that we are at a point where we want to refactor a lot of the API regarding the **kwargs that are scattered everywhere. The second main concern can come from this, and also maintenance of this in the long run. But both @sgugger and @LysandreJik have a better understanding on the specifics, will let them decide 🤗 

 @sgugger and @LysandreJik, I'm sorry to bother you.
What do you think about it? If the above discussion has cleared up any concerns, then I sincerely hope that you will consider making a positive decision.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.