ValueError: --optim adamw_torch_fused with --fp16 requires PyTorch>2.0

Imran_Ullah: <p>I install torch version 2. But again its give me the error</p>
<p>ValueError: --optim adamw_torch_fused with --fp16 requires PyTorch&gt;2.0</p>

ege_b: <p>Can you check the version of PyTorch imported on your script: <code>print(torch.__version__)</code>. Maybe you installed PyTorch 2.0 to your system/venv but are using venv/system python to run your script. Another thing to note is that the last stable version of PyTorch at the moment is 2.0.0. So are you getting the error PyTorch&gt;2.0 or PyTorch&gt;=2.0 ?</p>

Imran_Ullah: <p>Yeah , its show the pytorch version 2.0.0+cu117</p>

Imran_Ullah: <p>But they not working</p>
<p>Here is the code</p>
<pre><code class="lang-auto">import transformers
from datasets import load_dataset

datasets = data.map(
    tokenize, batched=True, remove_columns=data["train"].column_names
)

trainer = transformers.Trainer(
    model=model, 
    train_dataset=datasets['train'],
    args=transformers.TrainingArguments(
        per_device_train_batch_size=4, 
        gradient_accumulation_steps=4,
        num_train_epochs=3,
        torch_compile=True,
        optim="adamw_torch_fused",
        warmup_steps=100, 
        max_steps=200, 
        report_to = "tensorboard",
        learning_rate=2e-4, 
        fp16=True,
        logging_steps=1, 
        output_dir='PB7B'
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = False  # silence the warnings. Please re-enable for inference!
trainer.train()
</code></pre>
<p>For faster training I was add this two parameters</p>
<pre><code class="lang-auto">torch_compile=True,
optim="adamw_torch_fused",
</code></pre>

ege_b: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf2de8e9390e0ce51fb24a40693dabac4a5c684c.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf2de8e9390e0ce51fb24a40693dabac4a5c684c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf2de8e9390e0ce51fb24a40693dabac4a5c684c_2_690x449.png" alt="image" data-base62-sha1="tyNcfW7Fft3Vbtikw8MKau3LvdO" width="690" height="449" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf2de8e9390e0ce51fb24a40693dabac4a5c684c_2_690x449.png, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf2de8e9390e0ce51fb24a40693dabac4a5c684c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf2de8e9390e0ce51fb24a40693dabac4a5c684c.png 2x" data-dominant-color="F6F6EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">982×640 77.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As is says so here, you might need a nightly version of PyTorch, meaning the unstable master release which will explain the PyTorch &gt;2.0 error. I did not get a chance to try this yet, but it definitely is worthy of checking. Image is from <a href="https://huggingface.co/docs/transformers/v4.27.1/en/main_classes/trainer#transformers.TrainingArguments" class="inline-onebox" rel="noopener nofollow ugc">Trainer</a></p>

marksaroufim: <p>Is that error coming from HF trainer or from PyTorch? Regardless you can try casting your data to fp16 but I’d double check in HF forums what that error is about</p>

Imran_Ullah: <p>It’s come from hugging face trainer. He suggest that you should need to upgrade pytorch version grater then 2.</p>

ptrblck: <p>Related PR: <a href="https://github.com/huggingface/transformers/pull/22144" class="inline-onebox">[trainer] add `--optim adamw_torch_fused` for pt-2.0+ by stas00 · Pull Request #22144 · huggingface/transformers · GitHub</a></p>

Imran_Ullah: <p>They are not helpful. I have still the issues and can’t solve yet.</p>

ege_b: <aside class="quote no-group" data-username="Imran_Ullah" data-post="7" data-topic="175291">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/imran_ullah/48/58544_2.png" class="avatar"> Imran_Ullah:</div>
<blockquote>
<p>It’s come from hugging face trainer. He suggest that you should need to upgrade pytorch version grater then</p>
</blockquote>
</aside>
<p>well, did you update to the nightly release?</p>

Imran_Ullah: <p>Yeah,<br>
I just update to 2.0.0 using official nightly…</p>

marksaroufim: <p>The nightlies should start with 2.1.0 now - you can pick the selector here to get the right version <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a></p>

Imran_Ullah: <p>Thanks for answering <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=12" title=":hugs:" class="emoji" alt=":hugs:" loading="lazy" width="20" height="20"></p>