fix: Fixed CodeGenTokenizationTest::test_truncation failing test
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Fixed the following failing test
https://github.com/huggingface/transformers/blob/a27182b7fc57d07adc853689978649b2f6b1cf5b/tests/models/codegen/test_tokenization_codegen.py#L253

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker 

Comments:
> Thanks for handling @Sai-Suraj-27!
> 
> I'm assuming that this used to pass - it would be good if you were able to track down the offending commit which changed this behaviour

Thanks for the review @amyeroberts. The whole test function was added in this PR (https://github.com/huggingface/transformers/pull/17443). From [git blame](https://github.com/huggingface/transformers/blame/a27182b7fc57d07adc853689978649b2f6b1cf5b/tests/models/codegen/test_tokenization_codegen.py#L252-L262), I see that the test was not modified after this PR.

**Edit:** Sorry, I forgot to make the commit message such that it triggers the slow tests. 

> Thanks for the review @amyeroberts. The whole test function was added in this PR (https://github.com/huggingface/transformers/pull/17443). From [git blame](https://github.com/huggingface/transformers/blame/a27182b7fc57d07adc853689978649b2f6b1cf5b/tests/models/codegen/test_tokenization_codegen.py#L252-L262), I see that the test was not modified after this PR.

Ah, yes, sorry I wasn't clear. I don't mean necessarily the commit when the function was added / modified but the commit when the tests started failing. 

The way to do this is: 
* Checkout the commit when the function / model was first added. `git checkout d6b6fb9`
* Run the tests to see if they pass. If they don't they they've always been wrong and we can just fix like this. If they do pass, they we'll need to run git bisect. 
* Still on [d6b6fb9](https://github.com/huggingface/transformers/commit/d6b6fb9963e094216daa30ebf61224ca1c46921e), we init the git bisect and indicate we're on a good commit. `git bisect start` then `git bisect good`. 
* We then want to mark a known bad commit - so you can just checkout main `git checkout main` then `git bisect bad`. 
* From there, git will bisect the commit history to track down the offending commit. It will checkout in your local to a commit between the known good and bad commits. You should then run the tests. If they pass, you mark this commit as good with `git bisect good`, if they fail `git bisect bad`. You'll end up on the commit when this issue first occurred within a few iterations. The only issue will be if there's more than one offending commit between the initially marked good and bad ones. 

Let me know if this wasn't clear or if I can help at all 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32850). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> The way to do this is:
> 
>     * Checkout the commit when the function / model was first added. `git checkout d6b6fb9`
> 
>     * Run the tests to see if they pass. If they don't they they've always been wrong and we can just fix like this. If they do pass, they we'll need to run git bisect.
> 
>     * Still on [d6b6fb9](https://github.com/huggingface/transformers/commit/d6b6fb9963e094216daa30ebf61224ca1c46921e), we init the git bisect and indicate we're on a good commit. `git bisect start` then `git bisect good`.
> 
>     * We then want to mark a known bad commit - so you can just checkout main `git checkout main` then `git bisect bad`.
> 
>     * From there, git will bisect the commit history to track down the offending commit. It will checkout in your local to a commit between the known good and bad commits. You should then run the tests. If they pass, you mark this commit as good with `git bisect good`, if they fail `git bisect bad`. You'll end up on the commit when this issue first occurred within a few iterations. The only issue will be if there's more than one offending commit between the initially marked good and bad ones.
> 
> 
> Let me know if this wasn't clear or if I can help at all


Thanks for a such a detailed explanation @amyeroberts. I used `git bisect` to find the first bad commit that broke the test. It was initially passing [here](https://github.com/huggingface/transformers/pull/17443) in the PR.

![image](https://github.com/user-attachments/assets/be333459-73a2-48b5-9906-0c36597eb439)
So, this was the [PR](https://github.com/huggingface/transformers/pull/26570) that broke the test, I also confirmed this by running the test on the commit right before merging this PR which is [this one](https://github.com/huggingface/transformers/pull/26841), and the test was passing here.

Thanks for investigating @Sai-Suraj-27 and for finding the critical commit! 

OK, the change looks OK, let's just confirm with @ArthurZucker that this change in tokenization is expected / allowed following #26570