fix whisper re-compile
Author: jiqing-feng
Pull request: yes
Labels: name: Compilation

The code I removed will trigger re-compile when we use torch.compile. I just found the time complexity of the check is O(n), and the clamp op is also O(n). So we can just remove the check, in this case, we avoid torch re-compile and also have no extra computation.

To reproduce the error:
```python
import torch
import numpy as np
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="openai/whisper-small", torch_dtype=torch.float16, device_map="cuda:0")
inputs = np.random.rand(86699)

pipe.model.forward = torch.compile(pipe.model.forward, fullgraph=True)
pipe(inputs)
```

Traceback:
```
torch._dynamo.exc.UserError: Dynamic control flow is not supported at the moment. Please use functorch.experimental.control_flow.cond
to explicitly capture the control flow. For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/i
ndex.html#cond-operands

from user code:
   File "/workspace/jiqing/transformers/src/transformers/models/whisper/modeling_whisper.py", line 1779, in forward
    outputs = self.model(
  File "/workspace/jiqing/transformers/src/transformers/models/whisper/modeling_whisper.py", line 1630, in forward
    encoder_outputs = self.encoder(
  File "/workspace/jiqing/transformers/src/transformers/models/whisper/modeling_whisper.py", line 1072, in forward
    layer_outputs = encoder_layer(
  File "/workspace/jiqing/transformers/src/transformers/models/whisper/modeling_whisper.py", line 641, in forward
    torch.isinf(hidden_states).any() or torch.isnan(hidden_states).any()
```


Hi @SunMarc @ArthurZucker . Do you agree with that?

Comments:
cc @eustlb for whisper as well

Hi @Rocketknight1 @ArthurZucker , do you think it's ok?

In think this was added back then in order to allow fp16 training. Maybe we can instead check if self.training is enabled ? cc @ArthurZucker  https://github.com/huggingface/transformers/pull/10496 . If this works, we can modify all models that do this check 

> In think this was added back then in order to allow fp16 training. Maybe we can instead check if self.training is enabled ? cc @ArthurZucker #10496 . If this works, we can modify all models that do this check

I only considered inference.

Found a new regression that blocks torch.compile. Convert to draft until the regression is fixed.

Hi @ArthurZucker 
I test FP16 [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) on A100.


Here are latencys of eager and compile mode:

eager:
before change: 123ms
after change: 108ms

torch.compile
before change: 125ms
after change: 108ms

torch.compile with full_graph
before change: error
after change: 108ms

torch                      2.8.0.dev20250325+cu128

test script:
```python
import torch
import time
import numpy as np
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="openai/whisper-large-v2", torch_dtype=torch.float16, device_map="cuda:0")
inputs = np.random.rand(86699)

generation_config = pipe.model.generation_config
generation_config.cache_implementation="static"
generate_kwargs = {"generation_config": generation_config}


for _ in range(20):
    pipe(inputs, generate_kwargs=generate_kwargs)

for _ in range(5):
    start = time.time()
    pipe(inputs, generate_kwargs=generate_kwargs)
    end = time.time()
    print(f"eager pipeline latency: {(end-start)*1000} ms")

pipe.model.forward = torch.compile(pipe.model.forward)#, fullgraph=True)
for _ in range(20):
    pipe(inputs, generate_kwargs=generate_kwargs)

for _ in range(5):
    start = time.time()
    pipe(inputs, generate_kwargs=generate_kwargs)
    end = time.time()
    print(f"compile pipeline latency: {(end-start)*1000} ms")
```

The failed CI seems not related to my changes.