Add token type ids to CodeGenTokenizer
Author: st81
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #28098 

## Who can review?

@ArthurZucker @younesbelkada

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
This PR is WIP because I will check documentation update is needed and CodeGenTokenizer output includes `token_type_ids`. 

@ArthurZucker @younesbelkada
I've made the necessary changes to address the issue. Could someone please review the pull request at your earliest convenience? Your feedback is greatly appreciated.

Thank you!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29265). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks! You might have to update the `test_tokenizer_integration` as it will now return token_type_ids. Let's make sure slow tokenizer tests pass as well!

@ArthurZucker 

Thank you for taking the time to review the pull request! It seems that `test_tokenizer_integration` is missing from `tests/models/codegen/test_tokenization_codegen.py`. Would you like me to add `test_tokenizer_integration` to `tests/models/codegen/test_tokenization_codegen.py` similar to the other tokenizers tests?

> Thanks! You might have to update the `test_tokenizer_integration` as it will now return token_type_ids. Let's make sure slow tokenizer tests pass as well!

@ArthurZucker 
It seems that `test_tokenizer_integration` is missing from `tests/models/codegen/test_tokenization_codegen.py`. So I’ve added `test_tokenizer_integration`. Could you review again?

Will review tomorrow sorry ! 

I've addressed your comment and also added documentation for `CodeGenTokenizer.create_token_type_ids_from_sequences` ! Could you please review it again?

> Currently with the state of this PR:
> 
> ```python
> In [1]:  from transformers import AutoTokenizer
> 
> In [2]: tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-350M-mono")
> 
> In [3]:  tokenizer("Hey how are you")
> Out[3]: {'input_ids': [10814, 703, 389, 345], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]}
> ```
> 
> this is not backward compatible. `return_token_type_ids` should be set to false

@ArthurZucker 
I've updated `return_token_type_ids` to be False by default.

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-350M-mono")
>>> tokenizer("Hey how are you")
{'input_ids': [10814, 703, 389, 345], 'attention_mask': [1, 1, 1, 1]}
>>> tokenizer("Hey how are you", return_token_type_ids=True)
{'input_ids': [10814, 703, 389, 345], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]}
```


Cloud you review again?

> We have to change the default `return_token_type_ids` not in the test (it's alright to keep it) but most importantly in the **init** of both tokenizers, set it to ` self.return_token_type_ids = return_token_type_ids` and `return_token_type_ids=False` in the args!

@ArthurZucker 
Do you mean that we should make changes like below?

```python
class CodeGenTokenizer(PreTrainedTokenizer):
    def __init__(
        self,
        ...,
        return_token_type_ids=False,
        **kwargs,
    ):
        self.return_token_type_ids=return_token_type_ids
```

When looking at other tokenizers in the repository, I found that none of them have `return_token_type_ids` as an argument in the `__init__` function, and set `return_token_type_ids` as a class variable of the `tokenizer` like `self.return_token_type_ids = return_token_type_ids`. Would it be problematic from the perspective of library consistency to make `CodeGenTokenizer` the only one with such a specification?

Please let me know if there is any misunderstanding on my part.

@ArthurZucker 

Could you take a look at [this comment](https://github.com/huggingface/transformers/pull/29265#issuecomment-1983580444)? Your feedback is greatly appreciated.

@st81 Just to let you know @ArthurZucker is off for a week

> @st81 Just to let you know @ArthurZucker is off for a week

Thank you for letting me know about that! I'll wait for him to return.

@ArthurZucker 

> We have to change the default `return_token_type_ids` not in the test (it's alright to keep it) but most importantly in the **init** of both tokenizers, set it to ` self.return_token_type_ids = return_token_type_ids` and `return_token_type_ids=False` in the args!

I think current changes are fine to make the default `return_token_type_ids` is `False` because [it looks like the default value of `return_token_type_ids` is controlled by whether `self.model_input_names` contains `"token_type_ids"` or not](https://github.com/huggingface/transformers/blob/8dd4ce6f2cea1316cb5bd5ccbd348310ced17856/src/transformers/tokenization_utils_base.py#L3494-L3495). What are your thoughts on this?

> We have to change the default `return_token_type_ids` not in the test (it's alright to keep it) but most importantly in the **init** of both tokenizers, set it to ` self.return_token_type_ids = return_token_type_ids` and `return_token_type_ids=False` in the args!

I've made changes based on your [comment](https://github.com/huggingface/transformers/pull/29265#pullrequestreview-1921451566). Could you take a look?



Sorry for the delay, LGTM 

Thank you so much for reviewing!