Performance degradation on certain vision models from v4.51.*
Author: yuan-thomas
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.51.3
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.5.3
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.7.0a0+7c8ec84dab.nv25.03 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 3070 Laptop GPU

### Who can help?

vision models: @amyeroberts, @qubvel

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Run this script:
```py
from transformers import AutoImageProcessor, ConvNextV2Model
import torch
import torch.nn as nn
import time
from datasets import load_dataset

dataset = load_dataset("huggingface/cats-image")
image = dataset["test"]["image"][0]

image_processor = AutoImageProcessor.from_pretrained("facebook/convnextv2-large-1k-224")
model = ConvNextV2Model.from_pretrained("facebook/convnextv2-large-1k-224")

inputs = image_processor(image, return_tensors="pt")

start_time = time.time()

model.train()

logits = model(**inputs).last_hidden_state.mean(dim=1)  # [batch_size, hidden_size]
criterion = nn.BCEWithLogitsLoss()

fake_logits = torch.randn_like(logits)

loss = criterion(logits, fake_logits)
loss.backward()

print(time.time() - start_time)
```
### Expected behavior

It seems there is performance degradation between version 4.50.* and 4.51.*. Tried pytorch version 2.4 & 2.7.

In my testing, 4.51.* is about 4x slower than the previous version. Using the script attached:
- 4.50.* uses 1.1s
- 4.51.* uses 4.5s

Comments:
While duration is longer, GPU utilization is much higher too, from average of 30% in 4.50.* to 90% in 4.51.*, while running this script.

Hey @yuan-thomas, thanks for reporting the issue! I'm not sure if something has changed for the model between these versions. Have you tried benchmarking with a warm-up and at least several passes? I suppose a single pass might not be reliable enough. Thanks!

> Hey [@yuan-thomas](https://github.com/yuan-thomas), thanks for reporting the issue! I'm not sure if something has changed for the model between these versions. Have you tried benchmarking with a warm-up and at least several passes? I suppose a single pass might not be reliable enough. Thanks!

Hi @qubvel, thanks for the reply. Yes I have tried at least 20 epochs * 100 batches. It is very consistently slow in the new version. The code sample here is just for reproduction of issue. Anything other information that I can collect to further analyze?

Thanks!

@qubvel, @yuan-thomas  I have the same issue with 4.51, the inference is extremely slow with convnextv2, but it works fine with convnext, any updates has been made on certain models? (the inference code works well on 4.50, but its just way slower on the models with convnextv2 with 4.51)

Hey @Taehyoung-IVI @yuan-thomas, thank you both for reporting! Indeed, a bug exists; it was introduced with 
 - https://github.com/huggingface/transformers/pull/37041/files

Just made a fix here:
 - https://github.com/huggingface/transformers/pull/38015