rm input dtype change in CPU
Author: jiqing-feng
Pull request: yes

Hi @amyeroberts . Refer to [28199](https://github.com/huggingface/transformers/pull/28199). Since `Autocast` cannot integrate into the pipeline, I propose that keep the inputs dtype in the pipeline. Otherwise, it will block the low-precision usage in both ASR and text-to-audio. 

BTW, we will be ready for review once we confirm that it works on different CPUs, please keep this PR open. Thanks!

Comments:
Hi @amyeroberts . I have tested it on difference CPUs and didn't find any crashes. Could you please help me review it? Thx!

Hi @jiqing-feng, I think the correct solution here is to enable the necessary types in the ASR pipeline, rather than removing this cast.

> Hi @jiqing-feng, I think the correct solution here is to enable the necessary types in the ASR pipeline, rather than removing this cast.

Sorry for that I didn't make it clear. We can see that the function name is `_ensure_tensor_on_device`, so it should only ensure the tensor is on the correct device, instead of changing tensor types. Of course, we can enable the necessary types in the ASR pipeline, but not in this function.

@jiqing-feng Can you share a code snippet showing usage which this PR fixes and a full traceback of the error without this fix? 

> @jiqing-feng Can you share a code snippet showing usage which this PR fixes and a full traceback of the error without this fix?

Hi @amyeroberts . I run the following script in CPU:
```python
from transformers import pipeline
from datasets import load_dataset
from datasets import Audio
import torch

minds = load_dataset("PolyAI/minds14", name="de-DE", split="train")
minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
example = minds[0]

asr = pipeline("automatic-speech-recognition", model="maxidl/wav2vec2-large-xlsr-german", torch_dtype=torch.bfloat16)
output = asr(example["audio"]["array"])
print(output)
```
will get the following error:
```
Traceback (most recent call last):
  File "/home/jiqingfe/transformers/test.py", line 11, in <module>
    output = asr(example["audio"]["array"])                                                                                                                                                                                     File "/home/jiqingfe/transformers/src/transformers/pipelines/automatic_speech_recognition.py", line 292, in __call__
    return super().__call__(inputs, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/base.py", line 1154, in __call__                                                                                                                                   return next(
  File "/home/jiqingfe/transformers/src/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/pt_utils.py", line 266, in __next__                                                                                                                                processed = self.infer(next(self.iterator), **self.params)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/base.py", line 1068, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/automatic_speech_recognition.py", line 524, in _forward                                                                                                            outputs = self.model(**inputs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1967, in forward
    outputs = self.wav2vec2(
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1552, in forward
    extract_features = self.feature_extractor(input_values)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 460, in forward
    hidden_states = conv_layer(hidden_states)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 335, in forward
    hidden_states = self.conv(hidden_states)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 310, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 306, in _conv_forward
    return F.conv1d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.FloatTensor) and weight type (CPUBFloat16Type) should be the same or input should be a MKLDNN tensor and weight is a dense tensor
```

But it will output the correct answer if applied my changes:
```
{'text': 'ich möchte gerne geld auf mein konto einzallen'}
```

Hi @jiqing-feng, thanks for providing more information. This isn't something I think we should merge in. Not all CPUs support bf16 and so removing this will cause issues for users who are running on GPU for prediction. See #17637 for the reasons for adding this in. 

As mentioned in #28199, autocasting your model before passing to the pipeline is an alternative to adding autocast to the pipeline. 

The pipelines are not intended to cover all cases, but rather give a simple one-line entry point to predictions. I'd suggest using the [modeling code directly](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2ForCTC.forward.example), where you'll have more control 

> Hi @jiqing-feng, thanks for providing more information. This isn't something I think we should merge in. Not all CPUs support bf16 and so removing this will cause issues for users who are running on GPU for prediction. See #17637 for the reasons for adding this in.
> 
> As mentioned in #28199, autocasting your model before passing to the pipeline is an alternative to adding autocast to the pipeline.
> 
> The pipelines are not intended to cover all cases, but rather give a simple one-line entry point to predictions. I'd suggest using the [modeling code directly](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2ForCTC.forward.example), where you'll have more control

Hi @amyeroberts . Thanks for your reply. 

For the 1st issue, we know that not all CPUs support bf16 and fp16, but I think we should let it crash while forward so that the users will know that they shouldn't use low-precision data types, and they can cast model and inputs to fp32.

For the 2nd issue, I can't understand why it affects GPU. I ran the code in #17637 , and the device is `cuda:0`, so removing this code won't have any impact.


cc @Narsil 


@jiqing-feng Apologies - there was a typo on my part. I meant cause issues for users running on CPU for prediction. 

I understand the idea of allowing it to crash, but the reality is that this logic is a breaking change, as it will stop working for users when it previously did work. 

> @jiqing-feng Apologies - there was a typo on my part. I meant cause issues for users running on CPU for prediction.
> 
> I understand the idea of allowing it to crash, but the reality is that this logic is a breaking change, as it will stop working for users when it previously did work.

Sorry, I don't understand why it stops working for users. On my test, it won't work for bf16 models if we keep this code, and works if we remove this code. Can you give me an example to show how my changes impact users? Thx!

BTW, I added a warning there to remind users to use fp32 if the device does not support low-precision.

> > @jiqing-feng Apologies - there was a typo on my part. I meant cause issues for users running on CPU for prediction.
> > I understand the idea of allowing it to crash, but the reality is that this logic is a breaking change, as it will stop working for users when it previously did work.
> 
> Sorry, I don't understand why it stops working for users. On my test, it won't work for bf16 models if we keep this code, and works if we remove this code. Can you give me an example to show how my changes impact users? Thx!
> 
> BTW, I added a warning there to remind users to use fp32 if the device does not support low-precision.

Hi @amyeroberts . Do you mind having a look at my comment? Thx!

Hi @jiqing-feng, 

The reason it will break now it that previously users would rely on this casting to make their pipelines work. Most users use pipelines as a simple API to do a single call and get outputs, and therefore aren't aware of the internals. This will cause a sudden unexpected change in behaviour and is trying to solve a problem which no-one else has reported yet. 

I understand the desire to use bfloat16 for your use case when this dtype is supported on CPU. In this case, I would suggest using the modeling code directly instead of the pipeline. 

If bfloat16 becomes more widely supported on CPU or this becomes a commonly requested feature, then we can reconsider. For now, I don't think this PR should be merged in. 

> Hi @jiqing-feng,
> 
> The reason it will break now it that previously users would rely on this casting to make their pipelines work. Most users use pipelines as a simple API to do a single call and get outputs, and therefore aren't aware of the internals. This will cause a sudden unexpected change in behaviour and is trying to solve a problem which no-one else has reported yet.
> 
> I understand the desire to use bfloat16 for your use case when this dtype is supported on CPU. In this case, I would suggest using the modeling code directly instead of the pipeline.
> 
> If bfloat16 becomes more widely supported on CPU or this becomes a commonly requested feature, then we can reconsider. For now, I don't think this PR should be merged in.

@jiqing-feng , I think @amyeroberts 's concern is reasonable. Could you help to check whether your change will break current uses cases across precision. If you can show that your changes don't break anything and bring benefits to some of them, this will facilitate the discussion. 

> Hi @jiqing-feng,
> 
> The reason it will break now it that previously users would rely on this casting to make their pipelines work. Most users use pipelines as a simple API to do a single call and get outputs, and therefore aren't aware of the internals. This will cause a sudden unexpected change in behaviour and is trying to solve a problem which no-one else has reported yet.
> 
> I understand the desire to use bfloat16 for your use case when this dtype is supported on CPU. In this case, I would suggest using the modeling code directly instead of the pipeline.
> 
> If bfloat16 becomes more widely supported on CPU or this becomes a commonly requested feature, then we can reconsider. For now, I don't think this PR should be merged in.

Hi @amyeroberts . Thanks for your clarification. I understand your concern, so I tested my changes on CPU which only supports `fp32`, and my changes did not break anything. You can test my changes with any model in any CPU. If you still think it is not acceptable, I can follow your suggestion to cast input type in the ASR pipeline, see [here](https://github.com/huggingface/transformers/pull/28631#issuecomment-1906513738)

> > Hi @jiqing-feng,
> > The reason it will break now it that previously users would rely on this casting to make their pipelines work. Most users use pipelines as a simple API to do a single call and get outputs, and therefore aren't aware of the internals. This will cause a sudden unexpected change in behaviour and is trying to solve a problem which no-one else has reported yet.
> > I understand the desire to use bfloat16 for your use case when this dtype is supported on CPU. In this case, I would suggest using the modeling code directly instead of the pipeline.
> > If bfloat16 becomes more widely supported on CPU or this becomes a commonly requested feature, then we can reconsider. For now, I don't think this PR should be merged in.
> 
> Hi @amyeroberts . Thanks for your clarification. I understand your concern, so I tested my changes on CPU which only supports `fp32`, and my changes did not break anything. You can test my changes with any model in any CPU. If you still think it is not acceptable, I can follow your suggestion to cast input type in the ASR pipeline, see [here](https://github.com/huggingface/transformers/pull/28631#issuecomment-1906513738)

Hi @amyeroberts , should I enable casting inputs data type in the ASR pipeline? Would like to get your feedback, thx!

> > Hi @jiqing-feng,
> > The reason it will break now it that previously users would rely on this casting to make their pipelines work. Most users use pipelines as a simple API to do a single call and get outputs, and therefore aren't aware of the internals. This will cause a sudden unexpected change in behaviour and is trying to solve a problem which no-one else has reported yet.
> > I understand the desire to use bfloat16 for your use case when this dtype is supported on CPU. In this case, I would suggest using the modeling code directly instead of the pipeline.
> > If bfloat16 becomes more widely supported on CPU or this becomes a commonly requested feature, then we can reconsider. For now, I don't think this PR should be merged in.
> 
> Hi @amyeroberts . Thanks for your clarification. I understand your concern, so I tested my changes on CPU which only supports `fp32`, and my changes did not break anything. You can test my changes with any model in any CPU. If you still think it is not acceptable, I can follow your suggestion to cast input type in the ASR pipeline, see [here](https://github.com/huggingface/transformers/pull/28631#issuecomment-1906513738)

Can you share how you tested the pipelines here? Ideally with a reproducible snippet?

> > > Hi @jiqing-feng,
> > > The reason it will break now it that previously users would rely on this casting to make their pipelines work. Most users use pipelines as a simple API to do a single call and get outputs, and therefore aren't aware of the internals. This will cause a sudden unexpected change in behaviour and is trying to solve a problem which no-one else has reported yet.
> > > I understand the desire to use bfloat16 for your use case when this dtype is supported on CPU. In this case, I would suggest using the modeling code directly instead of the pipeline.
> > > If bfloat16 becomes more widely supported on CPU or this becomes a commonly requested feature, then we can reconsider. For now, I don't think this PR should be merged in.
> > 
> > 
> > Hi @amyeroberts . Thanks for your clarification. I understand your concern, so I tested my changes on CPU which only supports `fp32`, and my changes did not break anything. You can test my changes with any model in any CPU. If you still think it is not acceptable, I can follow your suggestion to cast input type in the ASR pipeline, see [here](https://github.com/huggingface/transformers/pull/28631#issuecomment-1906513738)
> 
> Can you share how you tested the pipelines here? Ideally with a reproducible snippet?

@jiqing-feng , I suppose talk on code and results is much better than talk-on-talk, could you pls share your test cases and results so people can discuss concretely.

We can use the format like:
test case 1  | test results before this PR | test results after this PR

Hi @amyeroberts . 
Here is my test script, it's derived from [asr_pipeline](https://huggingface.co/learn/audio-course/en/chapter2/asr_pipeline) and only changed the `torch_dtype` in the pipeline.
```python
from transformers import pipeline
from datasets import load_dataset
from datasets import Audio
import torch

minds = load_dataset("PolyAI/minds14", name="de-DE", split="train")
minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
example = minds[0]

asr = pipeline("automatic-speech-recognition", model="maxidl/wav2vec2-large-xlsr-german", torch_dtype=torch.bfloat16)
output = asr(example["audio"]["array"])
print(output)
```

### Experiments
My experiments configs and results before and after my change are as follows:
1. instance only support fp32, **torch_dtype=torch.float32**
    - before change: output `{'text': 'ich möchte gerne geld auf mein konto einzallen'}`
    - after change: output `{'text': 'ich möchte gerne geld auf mein konto einzallen'}`
2. instance only support fp32, **torch_dtype=torch.bfloat16**
    - before change: output `RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same`
    - after change: output `{'text': 'ich möchte gerne geld auf mein konto einzallen'}`
3. instance support bf16, **torch_dtype=torch.float32**
    - before change: output `{'text': 'ich möchte gerne geld auf mein konto einzallen'}`
    - after change: output `{'text': 'ich möchte gerne geld auf mein konto einzallen'}`
4. instance support bf16, **torch_dtype=torch.bfloat16**
    - before change: output `RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same`
    - after change: output `{'text': 'ich möchte gerne geld auf mein konto einzallen'}`


### Appending:
For config4, we can obtain 2x speed-up by adding `torch_dtype=torch.bfloat16` compared to fp32.
The full traceback of `RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same`:
```
Traceback (most recent call last):
  File "/home/jiqingfe/test_asr.py", line 11, in <module>
    output = asr(example["audio"]["array"])
  File "/home/jiqingfe/transformers/src/transformers/pipelines/automatic_speech_recognition.py", line 285, in __call__
    return super().__call__(inputs, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/base.py", line 1188, in __call__
    return next(
  File "/home/jiqingfe/transformers/src/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/base.py", line 1102, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/jiqingfe/transformers/src/transformers/pipelines/automatic_speech_recognition.py", line 524, in _forward
    outputs = self.model(**inputs)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1969, in forward
    outputs = self.wav2vec2(
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1554, in forward
    extract_features = self.feature_extractor(input_values)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 461, in forward
    hidden_states = conv_layer(hidden_states)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py", line 336, in forward
    hidden_states = self.conv(hidden_states)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 310, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/jiqingfe/.conda/envs/cpu/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 306, in _conv_forward
    return F.conv1d(input, weight, bias, self.stride,
RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same
```

The CPU config of the instance only supports fp32:
```
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              96
On-line CPU(s) list: 0-95
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz
Stepping:            7
CPU MHz:             2300.000
CPU max MHz:         3600.0000
CPU min MHz:         1000.0000
BogoMIPS:            4600.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-23,48-71
NUMA node1 CPU(s):   24-47,72-95
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
```

The CPU config of the instance supports bf16 (contains amx_bf16 and avx512_bf16):
```
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              224
On-line CPU(s) list: 0-223
Thread(s) per core:  2
Core(s) per socket:  56
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               143
Model name:          Intel(R) Xeon(R) Platinum 8480L
Stepping:            7
CPU MHz:             2000.000
CPU max MHz:         3800.0000
CPU min MHz:         800.0000
BogoMIPS:            4000.00
L1d cache:           48K
L1i cache:           32K
L2 cache:            2048K
L3 cache:            107520K
NUMA node0 CPU(s):   0-55,112-167
NUMA node1 CPU(s):   56-111,168-223
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
```

Hi @amyeroberts .  Pls let know if you need more cases : ) 

Hi @jiqing-feng - thanks for sharing! I won't have time to look this week, but will at the start of next week 

Hi @amyeroberts . Thanks for your review. I think it should be ready to merge : )

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28631). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.