Flash attention compilation warning?

Andrei_Fedorov: <p>Hello folks… can anyone advise why after upgrade to Pytorch 2.2.0 ( using pip in win10, RTX A2000 GPU) I am getting the following warning:<br>
AppData\Roaming\Python\Python311\site-packages\torch\nn\functional.py:5476: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at …\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)<br>
attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)</p>
<p>My code wasnt change, I use same as was using with torch 2.1</p>
<p>Release notes claims FlashAttention-v2 support, but due some reason pytorch wheel is no compiled with it… why ? any way to fix it ?<br>
In general after the warning all continue to work as usual, but the warning is annoiyng.</p>
<p>thanks</p>

GLZhu: <p>Add USE_FLASH_ATTENTION=1 in env</p>

Andrei_Fedorov: <p>unfortunately this didnt help… I set the env, but still have same warning<br>
I thinks<br>
this is not related to env of any other settings… the wheel was not compiled with FA support… so far it not supposed to run with any setting, isnt it ?</p>
<p>Meanwhile the code say that FA is available…</p>
<p>print(“Device name:”, torch.cuda.get_device_properties(‘cuda’).name)<br>
print(“FlashAttention available:”, torch.backends.cuda.flash_sdp_enabled())<br>
print(f’torch version: {torch.<strong>version</strong>}')</p>
<p>Device name: NVIDIA RTX A2000<br>
FlashAttention available: True<br>
torch version: 2.2.0+cu121</p>

GLZhu: <p>It seems that your wheel is just not compiled with flash attention.<br>
If you own the environment, just do:<br>
pip3 install --force-reinstall --pre torch torchtext torchvision torchaudio torchrec --extra-index-url <a href="https://download.pytorch.org/whl/nightly/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu121</a></p>

Andrei_Fedorov: <aside class="quote no-group" data-username="GLZhu" data-post="4" data-topic="196692">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/glzhu/48/57914_2.png" class="avatar"> GLZhu:</div>
<blockquote>
<p>pip3 install --force-reinstall --pre torch torchtext torchvision torchaudio torchrec --extra-index-url <a href="https://download.pytorch.org/whl/nightly/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu121</a></p>
</blockquote>
</aside>
<p>Have no idea what went wrong, but after such installation  and my code start I am getting<br>
AssertionError: Torch not compiled with CUDA enabled</p>
<p>rolling all back and wait stable release with complete FA support</p>
<p>thanks</p>

GLZhu: <aside class="quote no-group" data-username="Andrei_Fedorov" data-post="5" data-topic="196692">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_fedorov/48/66528_2.png" class="avatar"> Andrei_Fedorov:</div>
<blockquote>
<p>but after such installation and my code start I am getting<br>
AssertionError: Torch not compiled with CUDA enabled</p>
</blockquote>
</aside>
<p>You are already very close to the answer, try remove --pre in command above and install again</p>

Andrei_Fedorov: <p>thanks for you efforts… but unfortunately no progress<br>
I have installed<br>
while checking got<br>
Device name: NVIDIA RTX A2000<br>
FlashAttention available: True<br>
torch version: 2.3.0.dev20240122+cu121</p>
<p>but then, when I start my code I still got same warning \AppData\Roaming\Python\Python311\site-packages\torch\nn\functional.py:5504: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at …\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:308.)</p>
<p>guess the night build still compiled with no FlashAttention support…</p>

GLZhu: <p>Try compile your own pytorch then, follow the compile from source guide in pytorch repo with env USE_FLASH_ATTENTION=1 should suffice.</p>

Andrei_Fedorov: <p>thanks… just I think I am not such a hero yet  <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20">… no rush with FAv2. I’ll wait.</p>

ptrblck: <p>FlashAttentionV2 is not available on Windows yet. See <a href="https://github.com/pytorch/pytorch/issues/108175">#108175</a>.</p>

Andrei_Fedorov: <p>but is this complilation warning related to FAv2 in particular ?  or FA in general ? 2.1.2 has no this warning and seems that FA does work there.</p>

ptrblck: <p>I assume it’s specific to FlashAttentionV2, but also am not using Windows and thus cannot verify the support of previous implementations.<br>
Based on e.g. <a href="https://github.com/Dao-AILab/flash-attention/issues/345#issuecomment-1747473481">this comment</a> it seems to be the case.</p>