error " "baddbmm_with_gemm" not implemented for 'Half' " when running scaled_dot_product_attention

username: jasper_chen
cooked: <p>Hello, I’m new to Pytorch and learning through the documentation.<br>
The pytorch version is the CPU version on Windows 10 python 3. My laptop only has an Intel Iris graphic card.</p>
<p>When I was learning scaled dot product attention on:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html</a><br>
I was trying to reproduce the example on this page. Since my pytorch is not a cuda version so I typed the following on my Jupyter:</p>
<p>query = torch.rand(32, 8, 128, 64, dtype=torch.float16)<br>
key = torch.rand(32, 8, 128, 64, dtype=torch.float16)<br>
value = torch.rand(32, 8, 128, 64, dtype=torch.float16)<br>
atten1 = F.scaled_dot_product_attention(query,key,value)</p>
<p>And I get the following error:<br>
RuntimeError Traceback (most recent call last)<br>
~\AppData\Local\Temp\ipykernel_15000\311139239.py in<br>
2 key = torch.rand(32, 8, 128, 64, dtype=torch.float16)<br>
3 value = torch.rand(32, 8, 128, 64, dtype=torch.float16)<br>
----&gt; 4 atten1 = F.scaled_dot_product_attention(query,key,value)</p>
<p>RuntimeError: “baddbmm_with_gemm” not implemented for ‘Half’</p>
<p>Any help is greatly appreciated. Thanks!</p>
<h3><a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p>PyTorch version: 2.0.1+cpu<br>
Is debug build: False<br>
CUDA used to build PyTorch: None<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 Enterprise<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: N/A</p>
<p>Python version: 3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-10-10.0.19044-SP0<br>
Is CUDA available: False<br>
CUDA runtime version: No CUDA<br>
CUDA_MODULE_LOADING set to: N/A<br>
GPU models and configuration: No CUDA<br>
Nvidia driver version: No CUDA<br>
cuDNN version: No CUDA<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Architecture=9<br>
CurrentClockSpeed=2611<br>
DeviceID=CPU0<br>
Family=205<br>
L2CacheSize=5120<br>
L2CacheSpeed=<br>
Manufacturer=GenuineIntel<br>
MaxClockSpeed=2611<br>
Name=11th Gen Intel(R) Core™ i5-1145G7 @ 2.60GHz<br>
ProcessorType=3<br>
Revision=</p>
<p>Versions of relevant libraries:<br>
[pip3] flake8==4.0.1<br>
[pip3] mypy-extensions==0.4.3<br>
[pip3] numpy==1.21.5<br>
[pip3] numpydoc==1.4.0<br>
[pip3] torch==2.0.1<br>
[pip3] torchaudio==2.0.2<br>
[pip3] torchvision==0.15.2<br>
[conda] blas 1.0 mkl<br>
[conda] mkl 2021.4.0 haa95532_640<br>
[conda] mkl-service 2.4.0 py39h2bbff1b_0<br>
[conda] mkl_fft 1.3.1 py39h277e83a_0<br>
[conda] mkl_random 1.2.2 py39hf11a4ad_0<br>
[conda] numpy 1.21.5 py39h7a0a035_3<br>
[conda] numpy-base 1.21.5 py39hca35cd5_3<br>
[conda] numpydoc 1.4.0 py39haa95532_0<br>
[conda] torch 2.0.1 pypi_0 pypi<br>
[conda] torchaudio 2.0.2 pypi_0 pypi<br>
[conda] torchvision 0.15.2 pypi_0 pypi</p>

username: suraj.pt
cooked: <p>You need a GPU to use half precision (<code>torch.float16</code>). If you keep the <code>dtype=torch.float</code> (default) it will work fine.</p>

username: jasper_chen
cooked: <p>Hello Suraj,</p>
<p>Thanks for your reply! I noticed that I should specify:<br>
device = “cuda” if torch.cuda.is_available() else “cpu”<br>
and specify: device = device in the definition of query, key and value. After that, it works.</p>