Basic autocast usage

Mark_Hanslip: <p>Hi,</p>
<p>I’m a relative beginner compared to many on these fora, so kid gloves please…</p>
<p>I’ve created a Conda env with the latest PyTorch nightly build in a bid to try and use autocast to get a GAN working (<a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/relativistic_gan/relativistic_gan.py" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/relativistic_gan/relativistic_gan.py</a>) that currently causes memory errors in my setup (RTX 2070).</p>
<p>I’ve added the relevant code as per the docs - <span class="mention">@autocast</span>() to both forward passes, scaler = GradScaler() before training loop, and scaling on loss and backward pass inside the training loop. I’m just missing something basic like an import, or maybe I’ve downloaded the wrong nightly build (I used conda install pytorch torchvision cudatoolkit=10.1 -c pytorch-nightly).</p>
<p>Help please!</p>
<p>Thanks and apologies for luddite question,</p>
<p>Mark</p>

ptrblck: <p>What kind of error are you seeing and could you please post the PyTorch version you are using via <code>print(torch.__version__)</code>?</p>

Mark_Hanslip: <p>They’re very basic errors such as</p>
<pre><code class="lang-auto">NameError: name 'GradScaler' is not defined
</code></pre>
<p>and</p>
<pre><code class="lang-auto">NameError: name 'autocast' is not defined

</code></pre>
<pre><code class="lang-auto">current version is 1.6.0.dev20200407

</code></pre>
<pre><code class="lang-auto">
</code></pre>

mcarilli: <p><a class="mention" href="/u/mark_hanslip">@Mark_Hanslip</a>  Glad you’re trying the native API!  The full import paths are <code>torch.cuda.amp.autocast</code> and <code>torch.cuda.amp.GradScaler</code>.  Often, for brevity, usage snippets don’t show full import paths, silently assuming the names were imported earlier and that you skimmed the <a href="https://pytorch.org/docs/master/amp.html#torch.cuda.amp.autocast" rel="nofollow noopener">class or function declaration/header</a> to obtain each path.  For example, a snippet that shows</p>
<pre><code class="lang-auto">@autocast()
def forward...
</code></pre>
<p>silently assumes you wrote <code>from torch.cuda.amp import autocast</code> earlier in the script.</p>
<p>Try <code>from torch.cuda.amp import autocast</code> at the top of your script, or alternatively</p>
<pre><code class="lang-auto">@torch.cuda.amp.autocast()
def forward...
</code></pre>
<p>and treat GradScaler the same way.</p>
<p>The implicit-import-for-brevity-in-code-snippets is common practice throughout Pytorch docs, but may not be obvious if you’re relatively new to them.</p>
<p><strong>A separate concern is that the loss computation(s), in addition to the forward() methods, should run under autocast</strong> (for which you could use the context-manager option <code>with autocast()</code>).</p>
<p>The <a href="https://pytorch.org/docs/master/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers" rel="nofollow noopener">multi-model example</a> is likely relevant as well.  (<code>retain_graph</code> in the example has nothing to do with Amp, it’s present so the non-Amp parts shown are functionally correct, so ignore <code>retain_graph</code>.)</p>

Mark_Hanslip: <p><a class="mention" href="/u/mcarilli">@mcarilli</a> thanks so much. I’m more of a musician than programmer so sometimes basic things need clarifying.</p>
<p>I’m keeping an eye on GPU memory consumption with watch nvidia-smi, and there’s barely any difference with and without calling autocast on the model’s forward pass / scaling the loss computation etc.</p>
<p>I’ve tried to implement autocasting as follows, currently I’m hitting memory limits before even reaching loss calculation in the training loop:</p>
<p><span class="mention">@autocast</span>()<br>
def forward(self, z):<br>
out = self.l1(z)<br>
out = out.view(out.shape[0], 128, self.init_size, self.init_size)<br>
img = self.conv_blocks(out)<br>
return img</p>
<p>g_loss = adversarial_loss(discriminator(gen_imgs), valid)<br>
with autocast:<br>
scaler.scale(g_loss).backward()<br>
scaler.step(optimizer_G)<br>
scaler.update()</p>
<p>It feels as though I need to recast my inputs at the beginning of the training loop to FP16 (and possibly at the transforms/dataloader stage too?), is that right?</p>
<p>Mark</p>

shrutee: <p>Could you please post the GPU device name?</p>
<pre><code class="lang-auto">import torch
torch.cuda.get_device_name()
</code></pre>

Mark_Hanslip: <p>yes it’s ‘GeForce RTX 2070’</p>

shrutee: <p>Your forward pass and loss calculation should be inside autocast, and backward pass should be outside it.</p>
<p><code>g_loss = adversarial_loss(discriminator(gen_imgs), valid)</code><br>
should be inside autocast</p>
<pre><code class="lang-auto"># Creates model and optimizer in default precision
model = Net().cuda()
optimizer = optim.SGD(model.parameters(), ...)

for input, target in data:
    optimizer.zero_grad()

    # Enables autocasting for the forward pass (model + loss)
    with autocast():
        output = model(input)
        loss = loss_fn(output, target)

    # Exits the context manager before backward()
    loss.backward()
    optimizer.step()
</code></pre>

Sanjayvarma11: <p>Hi ptrblck i am facing an error while importing autocast.</p>
<pre><code class="lang-auto">My code is as follows:
from tqdm import tqdm_notebook, tnrange
import torch.nn.functional as F
import torch
from torch.cuda.amp import autocast 
class Train:
  def __init__(self, model, dataloader, optimizer, stats, scheduler=None, L1lambda = 0,criterion=None,use_amp=True):
    self.model = model
    self.dataloader = dataloader
    self.optimizer = optimizer
    self.scheduler = scheduler
    self.stats = stats
    self.L1lambda = L1lambda
    self.criterion=criterion
    self.loss1=0.0
    self.loss2=0.0
    self.loss=0.0
    self.use_amp=True
  def run(self):
    self.model.train()
    torch.cuda.empty_cache()
    pbar = tqdm_notebook(self.dataloader)
    for data1,data2,target1,target2 in pbar:
      # get samples
      data1,data2 = data1.to(self.model.device), data2.to(self.model.device)
      target1, target2 = target2.to(self.model.device), target2.to(self.model.device)
      self.optimizer.zero_grad()
      with autocast():
        output1,output2 = self.model(data1,data2)
        #print(target.shape)
        target1=target1.unsqueeze(1)
        target2=target2.unsqueeze(1)
        self.loss1=self.criterion[0](output1.float(), target1.half())
        self.loss2=self.criterion[1](output2.float(), target2.half())
        #print("loss1 {}".format(self.loss1))
        #print("loss2 {}".format(self.loss2))
        self.loss=(self.loss1+self.loss2)/2.0
    
      # In PyTorch, we need to set the gradients to zero before starting to do backpropragation because PyTorch accumulates the gradients on subsequent backward passes. 
      # Because of this, when you start your training loop, ideally you should zero out the gradients so that you do the parameter update correctly.

      # Predict
      
      #Implementing L1 regularization
      #print(self.loss)
      if self.L1lambda &gt; 0:
        reg_loss = 0.
        for param in self.model.parameters():
          reg_loss += torch.sum(param.abs())
        self.loss += self.L1lambda * reg_loss

      #print(self.loss)
      # Backpropagation
      self.loss.backward()
      #self.loss.backward()
      #self.optimizer.step()
Error is as follows:
ImportError: cannot import name 'autocast'

'''</code></pre>

ptrblck: <p><code>torch.cuda.amp</code> is available in the nightly binaries and the current master.<br>
If you are using an older version, you might need to update.</p>

Sanjayvarma11: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> hello sir i also posted another question can you please take a look into it.<br>
<a href="https://discuss.pytorch.org/t/22-5-gb-input-image-size-in-model-summary/81029">link</a></p>

Saurabh_Kataria: <p>Why are you not using gradient scaler? These commands should be used additionally?<br>
<code>scaler.scale(loss).backward()</code><br>
<code>scaler.step(optimizer)</code><br>
<code>scaler.update()</code></p>

Benjamin_Dubreu: <p>Hi guys,</p>
<p>I just upgraded my pytorch version to use AMP, and while Gradscaler now imports from torch.cuda.amp, I get the following error for autocast:</p>
<pre><code class="lang-auto">ImportError: cannot import name 'autocast' from 'torch.cuda.amp' (/home/benjamin/anaconda3/lib/python3.7/site-packages/torch/cuda/amp/__init__.py)
</code></pre>
<p>any idea why ?</p>

ptrblck: <p><code>autocast</code> is currently only available in the master branch and in the nightly binaries.<br>
If you want to try out mixed-precision training you would thus need to build from source or install the nightlies. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Benjamin_Dubreu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> oh my bad, I thought it was part of the library since it had been announced in beginning of May.</p>
<p>Just as a tip, to avoid bothering you in the future: where should I look to see what’s currently in the library and what is only in nightly ? Is there anything on <a href="https://pytorch.org/docs/master/amp.html#torch.cuda.amp.GradScaler" rel="nofollow noopener">this page</a> or elsewhere that would have given me this info without being annoying on the forums ?</p>
<p>Thanks for your answer !</p>

ptrblck: <p>The warning in the <a href="https://pytorch.org/docs/stable/amp.html">amp docs</a>, point towards the master/nightly builds for the complete mixed-precision training, but I see the confusion.<br>
Unfortunately, we couldn’t land it in 1.5, so both utilities will be available in the next stable release.</p>
<p>Also, don’t worry about asking here, as it’s not annoying at all and we are here for these questions. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Benjamin_Dubreu: <p>Ok thanks a lot for your answers !</p>

rishi_chandra: <p>Hi,<br>
Today in kaggle I installed the Nightly build through the command as mentioned on the <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> page: # conda install -y pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch-nightly -c conda-forge<br>
However again i am getting the error:</p>
<p>No module named ‘torch.cuda.amp.autocast’</p>
<p>As a second thought: It might also be an installation issue in Kaggle</p>

ptrblck: <p>That might be the case.<br>
Did you get any install logs in your Kaggle environment and if so, did you see which version was installed?<br>
Also, what does this return?</p>
<pre><code class="lang-python">import torch
print(torch.__version__)
print(torch.version.cuda)
print(torch.cuda.amp)
print(torch.cuda.amp.autocast)
</code></pre>