AudioClassificationPipelineTests::test_small_model_pt_fp16 fails for CUDA/XPU but passes for CPU
Author: dvrogozh
Pull request: no

With:
* https://github.com/huggingface/transformers/commit/92c5ca9dd70de3ade2af2eb835c96215cc50e815

On:
* Nvidia A10
* Intel Data Center GPU Max (PVC)

The commit https://github.com/huggingface/transformers/commit/f19135afc77053834f1b0cdf46d9a6bf7faf7cc3 introduced `AudioClassificationPipelineTests::test_small_model_pt_fp16` test which passes on CPU, but fails if running on CUDA or XPU. PR was:
* https://github.com/huggingface/transformers/pull/35435

Logs (for cuda):
```
python3 -m pytest tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16
===================================== test session starts ======================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/dvrogozh/git/huggingface/transformers
configfile: pyproject.toml
plugins: rich-0.2.0, xdist-3.6.1, asyncio-0.23.8, timeout-2.3.1
asyncio: mode=strict
collected 1 item

tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16 FAILED [100%]

=========================================== FAILURES ===========================================
__________________ AudioClassificationPipelineTests.test_small_model_pt_fp16 ___________________

self = <tests.pipelines.test_pipelines_audio_classification.AudioClassificationPipelineTests testMethod=test_small_model_pt_fp16>

    @require_torch
    def test_small_model_pt_fp16(self):
        model = "anton-l/wav2vec2-random-tiny-classifier"

        audio_classifier = pipeline("audio-classification", model=model, torch_dtype=torch.float16)

        audio = np.ones((8000,))
        output = audio_classifier(audio, top_k=4)

        EXPECTED_OUTPUT = [
            {"score": 0.0839, "label": "no"},
            {"score": 0.0837, "label": "go"},
            {"score": 0.0836, "label": "yes"},
            {"score": 0.0835, "label": "right"},
        ]
        EXPECTED_OUTPUT_PT_2 = [
            {"score": 0.0845, "label": "stop"},
            {"score": 0.0844, "label": "on"},
            {"score": 0.0841, "label": "right"},
            {"score": 0.0834, "label": "left"},
        ]
>       self.assertIn(nested_simplify(output, decimals=4), [EXPECTED_OUTPUT, EXPECTED_OUTPUT_PT_2])
E       AssertionError: [{'score': 0.0833, 'label': 'go'}, {'score': 0.0833, 'label': 'off'}, {'score': 0.0833, 'label': 'stop'}, {'score': 0.0833, 'label': 'on'}] not found in [[{'score': 0.0839, 'label': 'no'}, {'score': 0.0837, 'label': 'go'}, {'score': 0.0836, 'label': 'yes'}, {'score': 0.0835, 'label': 'right'}], [{'score': 0.0845, 'label': 'stop'}, {'score': 0.0844, 'label': 'on'}, {'score': 0.0841, 'label': 'right'}, {'score': 0.0834, 'label': 'left'}]]

tests/pipelines/test_pipelines_audio_classification.py:159: AssertionError
------------------------------------- Captured stderr call -------------------------------------
Device set to use cuda:0
======================================= warnings summary =======================================
tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16
  /home/dvrogozh/git/huggingface/transformers/src/transformers/configuration_utils.py:315: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================== short test summary info ====================================
FAILED tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16 - AssertionError: [{'score': 0.0833, 'label': 'go'}, {'score': 0.0833, 'label': 'off'}, {'sco...
================================= 1 failed, 1 warning in 1.99s =================================
```

Note: for XPU (upstream pytorch XPU, not IPEX) log is actually the same including same scores for labels

Does CUDA/XPU work correctly in this test? (I am confused seeing same scores `0.0833` for all 4 labels).

Overall, expectation is that test passes for CUDA/XPU or excluded for pytorch device backends if it's CPU specific.

CC: @jiqing-feng @ydshieh

Comments:
Hello,
I think this issue occurs because CUDA/XPU computation may lead to numerical instability when using torch.float16 due to lower precision compared to float32 or CPU computations as scores for every label is same i.e 0.833 try using mixed precision instead of torch.float16 or torch.float32.
you can also try using another model for prediction as "anton-l/wav2vec2-random-tiny-classifier" might not be optimized for torch.float16

Hi @dvrogozh . Thanks for your issue, could you please try this PR: [36359](https://github.com/huggingface/transformers/pull/36359) to check if it fix your issue?

It's indeed failing on our T4 runners. Thank you for pointing this out 👍