Add Gefen optimizer support to Trainer
Author: ndvbd
Pull request: yes

# What does this PR do?

Adds `optim="gefen"` support to `TrainingArguments` / `Trainer`.

Gefen is an AdamW-style memory-efficient optimizer. Gefen is a drop-in replacement for the AdamW optimizer for memory-efficient training. It keeps the familiar AdamW training recipe while dramatically reducing optimizer-state memory: an 8x reduction in AdamW memory footprint, or about 6.5 GiB saved per billion parameters, while maintaining AdamW-level performance. The reduced memory footprint lets you train larger models or use larger batch sizes and, as a result, achieve higher training throughput.

Users can now write:

```python
training_args = TrainingArguments(
    ...,
    optim="gefen",
)
```

Gefen-specific options can be passed via optim_args:

```
training_args = TrainingArguments(
    ...,
    optim="gefen",
    optim_args="fused=False,verbose=False",
)
```

Tests:

python -m pytest tests/trainer/test_trainer_optimizers.py::TrainerOptimizerIntegrationTest::test_gefen_trainer_adamw_args -q




Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46689)

The failing `pipelines_torch` job appears to have failed while downloading the hub cache, before tests ran:

- node 0: `curl: (35) TLS connect error: unexpected eof while reading`
- node 2: `curl: (56) Recv failure: Connection reset by peer`

The later `tests_output.txt` errors seem downstream of that missing test run. Could someone rerun the failed job?

Optimizer looks cool but we generally wait until they get significant adoption before we add them into `transformers`!

Thank @Rocketknight1 ! This sounds like a plan. We will update. 

The paper can be found here: https://arxiv.org/abs/2606.13894v1
Github code: https://github.com/ndvbd/Gefen

The code contains also fused-kernels path.

Hi @Rocketknight1 

Two recent updates:

1. Throughput experiments on a pretraining run of a 31.6B-parameter model based on a Nemotron-3 architecture (Mamba + MoE), using Megatron Bridge (NVIDIA's Megatron-Core training infrastructure), showed a 6× throughput improvement when moving from the 𝗔𝗱𝗮𝗺𝗪 optimizer (25 s/step, 86 tokens/s) to the 𝗚𝗲𝗳𝗲𝗻 optimizer (4.2 s/step, 520 tokens/s). This translates into millions of dollars in potential training cost savings. One reason for the speedup is that, with the Gefen optimizer, no CPU offloading was required, thereby eliminating CPU↔GPU communication overhead associated with optimizer state transfers. The throughput experiment, was conducted on 8 × H100 GPUs with 80 GiB of VRAM each. With the AdamW optimizer, approximately 385 GB of data had to be offloaded to CPU memory. In contrast, with the Gefen optimizer, no offloading was necessary, and all tensors resided entirely in GPU memory, utilizing a total of approximately 504 GiB of VRAM

2. A new a variant called Gefen-Muon was released as pip package: https://github.com/ndvbd/Gefen/
We took the same Gefen optimizer and added Muon orthagonalization into the first moment (only first moment is needed there), and you get the exact same performance as Muon with x4 less persistent memory. 

We believe HF users will benefit a lot from adding it to the platform at your earliest convenience. Other platform plan to add it too.

Thanks!

Usage first, then we'll add it, sorry! We've been burned too many times by optimizers and frameworks that look very promising but never really pan out. We take backward compatibility very seriously so once we put something in the library we're committing to maintaining it there indefinitely. That means we're quite conservative when it comes to adding frameworks and optimizers, but if it works as well as you say, I suspect you won't have trouble getting popular soon enough! :sweat_smile: 

@Rocketknight1 Thanks, sure, and what are you metrics for measuring usage? Github stars works?

Maintainers: recommend you hold off on this, the repo claims are not validated: 

[Issue #1 ](https://github.com/ndvbd/Gefen/issues/1#issuecomment-4802448972) (now deleted) highlighted gaps vs claims:

- broken FSDP (claimed working in repo/paper)
- missing paths for muon 1D matrix fallback (despite being touted as a "drop-in" replacement) 
- inability to reproduce VRAM savings and throughput increase claims
- learning rate parity claims  vs. adamw do not check out on modern models (e.g. qwen 3) 

FWIW - the repo authors seem to be banning people and deleting issues that counter their claims and manually pushing changes instead of merging PRs with proper author attribution as well.   

[gefen.log](https://github.com/user-attachments/files/29399939/gefen.log)

[adamw.log](https://github.com/user-attachments/files/29399940/adamw.log)

@thad0ctor has started a [fork](https://github.com/thad0ctor/Gefen-X) with some fixes that might be of use in the future. I agree with @the-DD5-is-too-damn-high in the meantime.

@iwr-redmond 
Hi, the second profile you mentioned is a fake account controlled by the former user you referred to. It was removed from the collaboration due to bot-driven activity, including automated pull requests generated by agents, as well as disrespectful language. We value constructive collaboration and respectful discussions, and we will address every issue.
As for HF, we are collecting more and more case-studies by users, and we will share them.
So far for pretraining we see good results both when changing from AdamW->Gefen, and Muon->GefenMuon.
Pretraining AdamW vs Gefen, training curve looks almost the same, and peak memory footprint goes down by x8: (GPT , openwebtext)

<img width="873" height="363" alt="image" src="https://github.com/user-attachments/assets/40b59727-ec94-4b3f-b2ae-3cf750cf4c21" />

Similar curves when pretraining larger models, and when swapping Muon->GefenMuon. Switching from Muon->GefenMuon, when Muon is trained with fp32 reduces persistent memory size by x4, and when Muon is in fp16, reduces footprint by x2.

<img width="872" height="388" alt="image" src="https://github.com/user-attachments/assets/7f04043f-329a-41be-b65e-72df9f5a5ce6" />


Although the main focus was pre-training, swapping Muon->GefenMuon also works for finetuning. For example, Qwen3B on Alpaca dataset, shows almost identical training curves, with less memory footprint.

<img width="876" height="366" alt="image" src="https://github.com/user-attachments/assets/9afdf8c2-581a-413d-aab3-25e08281f579" />

More feature-support is planned and we will update. Feel free to suggest ideas and feature requests, and thank you for the cooperation.



It would be helpful to (re)open issues in the Gefen/Gefen-X repositories so that the community can contribute their experiences and report any bugs that are encountered. I reckon that the previous round of reports have been quite helpful in clarifying the current use cases.