Switching between mixed-precision training and full-precision training after training is started

sdfadsfa: <p>Hi, using apex.amp or torch amp, is it possible that I switch between mixed precision training and full precision training after the training is started? For example, I might want the first 100 iterations to be trained with full precision, and switch to the mixed precision mode after the 100 iterations. Is this possible?</p>

ptrblck: <p>Yes, you could use the <code>enabled</code> arguments in <code>torch.cuda.amp.autocast</code> as well as in the creation of the <code>GradScaler</code> and switch them to <code>True/False</code> in your training run.</p>

sdfadsfa: <p>A quick question, during training when there’re frequent changes, say per 150 iterations, between mixed precision mode and the full precision mode (i.e. switching precision), why the accuracy gaps between purely full precision and the switching precision is so large? The accuracy gap between purely full precision and the purely mixed precision is not that large.</p>
<p>Also I’m not sure how to set GradScaler’s enabled=False dynamically. Isn’t that started before the training, and you can’t change it afterwards?</p>

ptrblck: <p>I don’t know what might be causing this effect as I haven’t seen or ran any experiments by frequently changing the precision.</p>
<aside class="quote no-group" data-username="sdfadsfa" data-post="3" data-topic="132366">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8c91f0/48.png" class="avatar"> sdfadsfa:</div>
<blockquote>
<p>Also I’m not sure how to set GradScaler’s enabled=False dynamically. Isn’t that started before the training, and you can’t change it afterwards?</p>
</blockquote>
</aside>
<p>Yes, usually you could either enable or disable amp in your script which can be done with the <code>enabled</code> arguments. If you want to dynamically switch the precision, you could use if conditions to pick the appropriate code path.</p>

sdfadsfa: <p>So for example, in the first 150 iterations I want full precision training, and in the later 150 iterations I want to switch back to mixed precision, and switching like this so on and so forth. The scaler is set before the training begin, i.e. <code>scaler = torch.cuda.amp.GradScaler(enabled=use_amp)</code>, then do you mean that during the training, say after 150 iterations, I need to re-initialize this scaler and change the <code>enabled</code> to <code>true</code> or <code>false</code>?</p>

ptrblck: <p>No, just skip the scaler usage if amp is disabled:</p>
<pre><code class="lang-auto">if use_amp:
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
else:
    loss.backward()
    ...
</code></pre>

sdfadsfa: <p>Thanks for the response! Now the torch amp can run correctly. Another quick question: how can I switch between <strong>Apex amp</strong> O0 and O1/O2/O3 during training time? Is model checkpointing and reloading the only way to accomplish this, since the <code>model, optimizer = amp.initialize(model, optimizer, opt_level=opt_level)</code> is set before the training begins?</p>

ptrblck: <p><code>apex.amp</code> is deprecated and you should use the native mixed precision util. via <code>torch.cuda.amp</code>.</p>