flash-attn cannot perform deterministic computation
Author: Ju-si-yuan
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.52.4
- Platform: Linux-5.10.134-010.ali5000.al8.x86_64-x86_64-with-glibc2.39
- Python version: 3.10.18
- Huggingface_hub version: 0.34.3
- Safetensors version: 0.6.1
- Accelerate version: 1.7.0
- Accelerate config:  not found
- DeepSpeed version: 0.16.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H800

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Using any training framework to perform training through Transformers.

Here, I am using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory/tree/v0.9.3 )
launch:
```bash
llamafactory-cli train examples/train_full/llama3_full_sft.yaml
```

Add the following content to llama3_full_sft.yaml:

full_determinism: True
flash_attn: fa2

### Expected behavior

After enabling deterministic computation, running twice produces inconsistent loss and gradients.

### Reason
Starting from version v2.4.1, flash-attn supports deterministic computation: https://github.com/Dao-AILab/flash-attention/pull/722. 
By default, the backward pass is non-deterministic because it uses `atomicAdd`. To enable deterministic behavior, you must set `deterministic=True` when calling `flash_attn_qkvpacked_func`. Transformers has also added corresponding support: https://github.com/huggingface/transformers/pull/31961.
The value of deterministic is controlled by `deterministic_g`, as shown in the following code:  
https://github.com/huggingface/transformers/blob/v4.51.2/src/transformers/modeling_flash_attention_utils.py#L276.
However, I believe there is an issue with this approach: `deterministic_g` is a global (module-level) variable that gets initialized as soon as the module is imported. This means that by the time `enable_full_determinism` is executed, `deterministic_g` has already been initialized to False. Therefore, the line at https://github.com/huggingface/transformers/blob/v4.51.2/src/transformers/trainer_utils.py#L79 does not affect the value of `deterministic_g`, resulting in it remaining permanently False.

Comments:
As shown in the figure below, right after I started training, the value of `deterministic_g` was already False.

<img width="3762" height="808" alt="Image" src="https://github.com/user-attachments/assets/e0f797ec-558c-4dbd-b214-97ae18de9880" />

cc @cyrilvallez for attention!

Can you check if #40002 fixes your issue? There shouldn't be a globally scoped variable anymore

I'm sorry, I see that it has already been fixed in versions after v4.53.0. This PR:  https://github.com/huggingface/transformers/pull/37185 .

We changed the fa integration quite a bit in v4.54 so it might not apply the same as in the most recent version(s). 

In both cases, it looks like it shouldn't be an issue anymore. So, I'll close this but let me know if this is still an issue.