Unexpected behaviour with transformers versions above 4.28 for Donut
Author: mdavudov
Pull request: no
Labels: name: bug

### System Info

Hello,

Big thanks to all the contributors on this repo! 

I would like to raise an issue, that was initially encountered when running example notebooks for Donut in Transformer Tutorials (https://github.com/NielsRogge/Transformers-Tutorials) by @NielsRogge . This is issue was previously raised on that repo, but the author advised to re-raise it here. Original issue: https://github.com/NielsRogge/Transformers-Tutorials/issues/496#issuecomment-2955991546

**Bug**:

The bug was encountered when trying to reproduce results from this notebook: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Donut/CORD/Fine_tune_Donut_on_a_custom_dataset_(CORD)_with_PyTorch_Lightning.ipynb

When using newer versions of `transformers` there is strange behaviour during training, as the model shows much higher validation edit distance values than expected. This is fixed by downgrading to versions `4.28.1` or `4.25`. 

Reference code uses the following classes from `transformers`: 

- `DonutProcessor`
- `VisionEncoderDecoderModel`
- `VisionEncoderDecoderConfig`

The difference can be seen on the attached screenshot, where the red line shows validation edit distance metric when running on `4.28.1` and the orange one when running on `4.36.0`.

Was there any changes introduced after `4.28.1` that could be causing it, and are there any known ways of fixing them? 

<img width="3026" height="356" alt="Image" src="https://github.com/user-attachments/assets/6b209a3c-3dd9-4a3a-9b45-a339d381cac7" />

**Environment**

Output of `transformers env` for `4.28.1`:

```
- `transformers` version: 4.28.1
- Platform: Linux-6.1.134-152.225.amzn2023.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.12
- Huggingface_hub version: 0.32.4
- Safetensors version: 0.5.3
- PyTorch version (GPU?): 2.7.1+cu128 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO
```

for `4.36.0` (version where issue is encountered):
```
- `transformers` version: 4.36.0
- Platform: Linux-6.1.134-152.225.amzn2023.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.12
- Huggingface_hub version: 0.32.4
- Safetensors version: 0.5.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.7.1+cu128 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO
```

Thank you for you time, and please let me know what I can do on my end to make it easier to diagnose the issue more precisely. 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The bug was encountered when trying to reproduce results from this notebook: 

https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Donut/CORD/Fine_tune_Donut_on_a_custom_dataset_(CORD)_with_PyTorch_Lightning.ipynb

To reproduce: 

1. Follow the notebook as-is, this will install the latest version of transformers
2. Continue until the training step and run the training
3. Observe unexpectedly high validation edit distance metrics

To fix:

1. Pin the transformers version to `4.28.1`
2. Run the notebook again
3. You should observe a much lower validation edit distance metrics

### Expected behavior

I expect the training behaviour to be similar on newer versions of `transformers` and the performance not to degrade so drastically.

Comments:
Hi @mdavudov, I'm not sure why this happened! If you could run a [git bisect](https://www.git-tower.com/learn/git/faq/git-bisect) for us it would be very helpful, to identify the exact commit where the issue appears. 

I'm running into the same issue, although in my case it happened with a different version than the one @mdavudov mentioned. When I train the Donut model on version 4.40, everything works as expected, but with the latest version of Transformers, I get the same unexpected results. I spent a few days trying to figure out what I was doing wrong before realizing it was likely a version-related problem. I was actually about to open a new issue when I came across this thread.

It would be great to be able to use the latest version since it includes important performance improvements.

Hi! Sorry for the late response.

It's taking a while to rerun things and solve other dependency issues along the way. I will get there, but so far narrowed down the issue to this range:

from 6ecb2ab67971a13e9832c2df84f6c9d1e7e0dbf0 
to c48787f347bd (4.36)

Will continue and send an update when identify the smallest range

I tested it on multiple versions with same code and dataset. Until version `4.48.0` I get good results, on version `4.49.0` the code breaks and on version `4.50.0` I get bad results:

<img width="1150" height="792" alt="Image" src="https://github.com/user-attachments/assets/fe656b77-bdf1-4d6f-9106-3f9fc5430454" />

I hope it helps.

cc @zucchini-nlp FYI, Donut fine-tuning doesn't work anymore since v 4.49

> If you could run a [git bisect](https://www.git-tower.com/learn/git/faq/git-bisect) for us it would be very helpful, to identify the exact commit where the issue appears.

+++

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I ran two separate `git bisect` sessions:

- One to find the commit that introduced the bug causing Donut training to fail on version `4.49.0`.

- Another to find the commit that fixed the error (to check if, after fixing the bug, the training shows unexpected behavior).

Bisect to find bug:

```
# bad: [a22a4378d97d06b7a1d9abad6e0086d30fdea199] v4.49.0
# good: [6bc0fbcfa7acb6ac4937e7456a76c2f7975fefec] [WIP] Emu3: add model (#33770)
git bisect start 'v4.49.0' 'v4.48.0'
# good: [8de7b1ba8d126a6fc9f9bcc3173a71b46f0c3601] Add flex_attn to diffllama (#35601)
git bisect good 8de7b1ba8d126a6fc9f9bcc3173a71b46f0c3601
# bad: [692afa102d202716f0073cd54c07c78a25eebb07] Update `squad_convert_example_to_features` to work with numpy v2 (#35955)
git bisect bad 692afa102d202716f0073cd54c07c78a25eebb07
# bad: [ec97417827471351c86cf655ee94d38ddf5bfa3f] Don't import torch.distributed when it's not available (#35777)
git bisect bad ec97417827471351c86cf655ee94d38ddf5bfa3f
# good: [fd4f14c968cf186a4dec325143cc89c10ca6dc5e] Fix: Falcon tie_word_embeddings in GGUF (#35715)
git bisect good fd4f14c968cf186a4dec325143cc89c10ca6dc5e
# bad: [94ae9a8da16b13cc28fa29201008c1c08119a2fc] OwlViT/Owlv2 post processing standardization (#34929)
git bisect bad 94ae9a8da16b13cc28fa29201008c1c08119a2fc
# bad: [80dbbd103c217f422de91a3265bf6d8e8bc414f7] 🧹 remove `generate`-related objects and methods scheduled for removal in v4.48 (#35677)
git bisect bad 80dbbd103c217f422de91a3265bf6d8e8bc414f7
# bad: [1302c32a844d737154f21a553a7e54fbb41301e4] Add proper jinja2 error (#35533)
git bisect bad 1302c32a844d737154f21a553a7e54fbb41301e4
# bad: [8b78d9d6e797bc842b4c1364105b7741371a729c] Fix the bug that `Trainer` cannot correctly call `torch_jit_model_eval` (#35722)
git bisect bad 8b78d9d6e797bc842b4c1364105b7741371a729c
# bad: [2cbcc5877d664f1d227cac419531cd2a78ddbd4b] Fix condition when GA loss bug fix is not performed (#35651)
git bisect bad 2cbcc5877d664f1d227cac419531cd2a78ddbd4b
# first bad commit: [2cbcc5877d664f1d227cac419531cd2a78ddbd4b] Fix condition when GA loss bug fix is not performed (#35651)
```

Bisect to find bug fix:
("good" = still raises error; "bad" = unexpected behavior)

```
git bisect start
# status: waiting for both good and bad commits
# good: [a22a4378d97d06b7a1d9abad6e0086d30fdea199] v4.49.0
git bisect good a22a4378d97d06b7a1d9abad6e0086d30fdea199
# status: waiting for bad commit, 1 good commit known
# bad: [0b057e66b52556da3a1cbc29e2a98c0784ea9c33] fix  import issue
git bisect bad 0b057e66b52556da3a1cbc29e2a98c0784ea9c33
# good: [bcfc9d795e1330faaa8b39ffa18732f8b40fe7c0] [Bugfix] Fix reloading of pixtral/llava configs (#36077)
git bisect good bcfc9d795e1330faaa8b39ffa18732f8b40fe7c0
# good: [e9756cdbc7dcec91ea0dde55c165d6276bd08252] [docs] Serving LLMs (#36522)
git bisect good e9756cdbc7dcec91ea0dde55c165d6276bd08252
# good: [3017536ebf3f215130cf4925a574c19fb4854cae] fix hqq due to recent modeling changes (#36771)
git bisect good 3017536ebf3f215130cf4925a574c19fb4854cae
# bad: [8e97b44087341837a2a9503fed9469c3abfd9eb8] [modular] Sort modular skips (#36304)
git bisect bad 8e97b44087341837a2a9503fed9469c3abfd9eb8
# good: [63c31165306226798c1353ca93701c7e11ce4e4f] One more fix for reviewer assignment (#36829)
git bisect good 63c31165306226798c1353ca93701c7e11ce4e4f
# good: [94555437e277949e6b3be9e97317df2a5e046d95] Disable inductor config setter by default (#36608)
git bisect good 94555437e277949e6b3be9e97317df2a5e046d95
# bad: [e7337ee7be682c9601cc1bb2b189759e9bfb104e] Pass num_items_in_batch directly to loss computation (#36753)
git bisect bad e7337ee7be682c9601cc1bb2b189759e9bfb104e
# good: [3f03c379d2b48fa5f84b907599211098d95d670a] fix tiktoken convert to pass AddedToken to Tokenizer (#36566)
git bisect good 3f03c379d2b48fa5f84b907599211098d95d670a
# good: [8b479e39bb677df7387834b23795dc8b68af8866] Saving `Trainer.collator.tokenizer` in when `Trainer.processing_class` is `None` (#36552)
git bisect good 8b479e39bb677df7387834b23795dc8b68af8866
# first bad commit: [e7337ee7be682c9601cc1bb2b189759e9bfb104e] Pass num_items_in_batch directly to loss computation (#36753)
```

Training comparison between the commit before the bug (`fd4f14c968cf186a4dec325143cc89c10ca6dc5e`) and after the bug fix (`e7337ee7be682c9601cc1bb2b189759e9bfb104e`):

<img width="1326" height="399" alt="Image" src="https://github.com/user-attachments/assets/e5ecaa18-5548-4b7a-976d-aad45a1e1a13" />

It looks like the problem was introduced between these two commits, but I can’t determine exactly which one, since I can’t run the training on any commit in that range.

Note: I’m running the training with a small dataset, and I consistently get accuracy below 0.2 for the unexpected behavior, compared to around 0.45 before it.


Looking at the bug fix commit (`e7337ee7be682c9601cc1bb2b189759e9bfb104e`), it seems it also modifies the Donut loss function:

<img width="1293" height="928" alt="Image" src="https://github.com/user-attachments/assets/09bbb1e9-fd28-4028-9553-0b61ff8eef06" />

I tried applying the bug fix to commit `8b479e39bb677df7387834b23795dc8b68af8866` and running the training again. Here are the results (red):

<img width="1330" height="399" alt="Image" src="https://github.com/user-attachments/assets/f5aa97ba-927a-47f7-b327-f0e15cc0bb8f" />

The issue seems to be related to the updated loss function.

I tried overwriting the loss_function on commit `e7337ee7be682c9601cc1bb2b189759e9bfb104e` with the same logic that was used before the changes:

```python
def loss_function(
    logits: torch.Tensor,
    labels: torch.Tensor,
    vocab_size: int,
    num_items_in_batch: int,
) -> torch.Tensor:
    loss_fct = CrossEntropyLoss()
    loss = loss_fct(logits.reshape(-1, vocab_size), labels.reshape(-1))
    return loss

encoder_decoder_config = VisionEncoderDecoderConfig.from_pretrained(config["pretrained_model_name_or_path"])
model: VisionEncoderDecoderModel = VisionEncoderDecoderModel.from_pretrained(
    config["pretrained_model_name_or_path"],
    config=encoder_decoder_config,
)
model.loss_function = loss_function
```

…and I got these training results (orange):

<img width="1327" height="397" alt="Image" src="https://github.com/user-attachments/assets/8db5e195-17c9-4ceb-84d8-2ca40072bc22" />

The issue seems to be caused by the `ForCausalLMLoss` function that is used by default.

Thanks a lot for looking into this! So in short, https://github.com/huggingface/transformers/pull/36753 seems to have introduced non-equivalent behaviour?

Based on comparing these 2, it looks like the `ForCausalLMLoss` shifts the labels one position?

@jgabriellacerda Great job on finding the faulty PR. Since you already have a tiny training script set-up, can you also verify if setting `accepts_loss_kwargs = False` fixes the issue, smth similar to https://github.com/huggingface/transformers/pull/37208/files? If not, I will tag someone from Trainer maintainers to take a look

If that fixes the issue, a PR would be very welcome

Does this issue only occur using the Trainer class, or also with native PyTorch training?

Hi all! So as I understood from the conversation, now the only way to run the code and get proper results is it to downgrade `transformers`? Apart from the fix on the loss function?

Yes at the moment you'd need to downgrade. Pinging @SunMarc here as this seems related to recent updates on GA and Trainer

> [@jgabriellacerda](https://github.com/jgabriellacerda) Great job on finding the faulty PR. Since you already have a tiny training script set-up, can you also verify if setting `accepts_loss_kwargs = False` fixes the issue, smth similar to https://github.com/huggingface/transformers/pull/37208/files? If not, I will tag someone from Trainer maintainers to take a look
> 
> If that fixes the issue, a PR would be very welcome


@zucchini-nlp I tried setting `accepts_loss_kwargs = False` on `VisionEncoderDecoderModel`, and got the same bad results.
Here are the metrics:


<img width="1478" height="399" alt="Image" src="https://github.com/user-attachments/assets/607d0da9-2c23-46c0-b899-cfed28975de1" />


- Orange: no changes
- Green: `accepts_loss_kwargs = False`
- Pink: overwriting `loss_function`

All runs used commit `ffcb344612a29da2f3c35c6f12877b50e67e4c87`.

> > [@jgabriellacerda](https://github.com/jgabriellacerda) Great job on finding the faulty PR. Since you already have a tiny training script set-up, can you also verify if setting `accepts_loss_kwargs = False` fixes the issue, smth similar to https://github.com/huggingface/transformers/pull/37208/files? If not, I will tag someone from Trainer maintainers to take a look
> > If that fixes the issue, a PR would be very welcome
> 
> [@zucchini-nlp](https://github.com/zucchini-nlp) I tried setting `accepts_loss_kwargs = False` on `VisionEncoderDecoderModel`, and got the same bad results. Here are the metrics:
> 
> <img alt="Image" width="1478" height="399" src="https://private-user-images.githubusercontent.com/49662646/487405534-607d0da9-2c23-46c0-b899-cfed28975de1.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTc1MDIyNDcsIm5iZiI6MTc1NzUwMTk0NywicGF0aCI6Ii80OTY2MjY0Ni80ODc0MDU1MzQtNjA3ZDBkYTktMmMyMy00NmMwLWI4OTktY2ZlZDI4OTc1ZGUxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwOTEwVDEwNTkwN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ3MDgwNjJiZTdiZTE1Y2RkY2NhZWY0OGEyY2E3YmEyMDE4MjM2YTRmZTNiMGVkMGQ2MjE0MTg4MDU0MzNlNmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.x4_3T0pmfnhP-K3ddkYyvq1fNmG_jNBXleOH2LzzTeE">
> * Orange: no changes
> * Green: `accepts_loss_kwargs = False`
> * Pink: overwriting `loss_function`
> 
> All runs used commit `ffcb344612a29da2f3c35c6f12877b50e67e4c87`.

@jgabriellacerda So did Loss function overwritting work? I would also like to take a look at the bug, but a bit lost in the conversation :)

@katarzynasornat Yes, it looks like overwriting the `loss_function` to use the old implementation works as a workaround for any version ≥ `4.50.0`.
Commit `e7337ee7be682c9601cc1bb2b189759e9bfb104e` changed `VisionEncoderDecoderModel`’s loss from a hardcoded implementation to [ForCausalLMLoss](https://github.com/huggingface/transformers/blob/6eb32558424c6a461252769f7aca3b3de834912b/src/transformers/loss/loss_utils.py#L45), which adds extra logic like label shifting. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.