Fix DeepSpeed mixed precision precedence over Accelerate defaults
Author: notkisk
Pull request: yes

## Summary

Fixes issue [[#39849](https://github.com/huggingface/transformers/issues/39849)] where Accelerate would default to `bf16` mixed precision even when a DeepSpeed config specifies `fp16`, causing the following error:

> `ValueError: --mixed_precision arg cannot be set to bf16 when fp16 is set in the DeepSpeed config file.`

This PR ensures that DeepSpeed configuration takes precedence over `TrainingArguments` defaults while preserving explicit user settings.

---

## Root Cause

The issue was caused by the initialization order in `TrainingArguments.__post_init__()`. The `ACCELERATE_MIXED_PRECISION` environment variable was being set **before** the DeepSpeed config was processed, preventing it from overriding Accelerate’s defaults.

---

## Changes Made

### 1. Added DeepSpeed Config Override Logic

* Added `override_training_args_from_deepspeed()` method to `HfTrainerDeepSpeedConfig` class.
* This method checks DeepSpeed config for `fp16`/`bf16` settings and overrides `TrainingArguments` defaults accordingly.
* Explicit user choices are preserved, but DeepSpeed config can override defaults if no user input is provided.

### 2. Fixed Initialization Order

* Moved the mixed precision environment variable setting in `TrainingArguments.__post_init__()` to occur **after** DeepSpeed config processing.
* Ensures DeepSpeed config overrides are applied **before** environment variables are set.

---

## Behavior

The fix enforces the following **precedence hierarchy**:

1. **Explicit user settings** – Highest priority
   E.g., `fp16=True` or `bf16=True` passed by user.
2. **DeepSpeed config** – Medium priority
   E.g., `"fp16": {"enabled": true}` or `"bf16": {"enabled": true}` in config file.
3. **TrainingArguments defaults** – Lowest priority

---

## Test Plan

* ✅ Verified the original reproduction case no longer fails.
* ✅ Tested that DeepSpeed `fp16` config overrides default correctly.
* ✅ Tested that DeepSpeed `bf16` config overrides default correctly.
* ✅ Confirmed explicit user settings take precedence over DeepSpeed config.
* ✅ Ensured environment variables are set correctly in all scenarios.
* ✅ Ran existing DeepSpeed test suite to check for regressions.
* ✅ Rebased on latest `main` and verified fix still works.

---

## Files Modified

* `src/transformers/integrations/deepspeed.py` – Added override logic and method call.
* `src/transformers/training_args.py` – Reordered mixed precision env var setup.

---

## Branch Info

* **PR Branch:** `fix-deepspeed-mixed-precision-precedence` (rebased on latest `main`)
* **Base Branch:** `main`


Comments:
@zucchini-nlp 

cc @sunmarc

@Rocketknight1 @ydshieh @gante @ArthurZucker 🙌

Hey, can you add some tests for this behaviour?

cc @S1ro1 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39856). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc 

> Thanks for the PR. Did you manage to reproduce the initial error from the PR @notkisk ? cc @S1ro1 for visibility

as far as i remember, yes i did reproduce the issue and the pr fixed it

Well, the issue is that we shouldn't overwrite training_args default. Hence i'm reverting this PR. The issue with the original issue was that mixed_precision was set to bf16 somehow