xpu: test_eager_matches_sdpa_inference tests fail with pytorch XPU backend
Author: dvrogozh
Pull request: no
Labels: name: Tests

name: bug

With:
* https://github.com/huggingface/transformers/commit/54be2d7ae87e873482b984cc956e165ca4dc0ba3
* https://github.com/huggingface/accelerate/commit/e11d3ceff3a49378796cdff5b466586d877d5c60
* https://github.com/pytorch/pytorch/commit/e429a3b72e787ddcc26ee2ba177643c9177bab24

```
$ cat spec.py
import torch
DEVICE_NAME = 'xpu'
MANUAL_SEED_FN = torch.xpu.manual_seed
EMPTY_CACHE_FN = torch.xpu.empty_cache
DEVICE_COUNT_FN = torch.xpu.device_count

$ TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest --pspec tests/models -k test_eager_matches_sdpa_inference
<...>
FAILED tests/models/audio_spectrogram_transformer/test_modeling_audio_spectrogram_transformer.py::
    Here we also overwrite some of the tests of test_modeling_common.py, as AST does not use input_ids, inputs_embeds,
    attention_mask and seq_length.
    ::test_eager_matches_sdpa_inference_0_float16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 4.739e-05,...
FAILED tests/models/audio_spectrogram_transformer/test_modeling_audio_spectrogram_transformer.py::
    Here we also overwrite some of the tests of test_modeling_common.py, as AST does not use input_ids, inputs_embeds,
    attention_mask and seq_length.
    ::test_eager_matches_sdpa_inference_1_bfloat16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 5.913e-04,...
FAILED tests/models/bart/test_modeling_bart.py::BartModelTest::test_eager_matches_sdpa_inference_0_float16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 7.510e-06,...
FAILED tests/models/bart/test_modeling_bart.py::BartModelTest::test_eager_matches_sdpa_inference_1_bfloat16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 7.772e-05,...
FAILED tests/models/bart/test_modeling_bart.py::BartStandaloneDecoderModelTest::test_eager_matches_sdpa_inference_0_float16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 2.402e-05,...
FAILED tests/models/bart/test_modeling_bart.py::BartStandaloneDecoderModelTest::test_eager_matches_sdpa_inference_1_bfloat16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 3.490e-04,...
FAILED tests/models/bert/test_modeling_bert.py::BertModelTest::test_eager_matches_sdpa_inference_0_float16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 5.555e-05,...
FAILED tests/models/bert/test_modeling_bert.py::BertModelTest::test_eager_matches_sdpa_inference_1_bfloat16 - AssertionError: False is not true : padding_side=left, use_mask=False, enable_kernels=False: mean relative difference: 3.567e-04,...
<...>
======================= 159 failed, 89 passed, 793 skipped, 75366 deselected, 319 warnings in 74.89s (0:01:14) =======================
```

CC: @amyeroberts @ydshieh 

Comments:
Please, help review PR with the fix:
* #34889