Converting model into 16 points precisoin (float16) instead of 32

Karan_Chhabra: <p>Hi,<br>
I am trying to train the model on mixed precision, so for the same I am using the command:<br>
<code>model.half()</code><br>
But I am getting the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/7724e9c9cf344b99aafad73e478166e81375b25c.png" data-download-href="https://discuss.pytorch.org/uploads/default/7724e9c9cf344b99aafad73e478166e81375b25c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/7724e9c9cf344b99aafad73e478166e81375b25c.png" alt="image" data-base62-sha1="gZZU7GHBkjR5pFfespjhverNFBq" width="690" height="213" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1363×422 27.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So when I convet my input and labels also to half but it seem like the error is caused due to the line:</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>I have tried to convert loss back into floating point value and run the same, but still I am getting the following error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff180ed1a24055cac4cb1a0ca8b29eb61b03a34f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff180ed1a24055cac4cb1a0ca8b29eb61b03a34f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff180ed1a24055cac4cb1a0ca8b29eb61b03a34f.png" alt="image" data-base62-sha1="AoFiwo2hz3ysqioj0HZ7dV3FlH9" width="690" height="202" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1383×405 41.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any suggetions?</p>

ptrblck: <p>Calling <code>model.half()</code> would not train the model in mixed-precision, but half precision.<br>
Automatic mixed-precision can be used via <a href="https://pytorch.org/docs/stable/amp.html"><code>torch.cuda.amp</code></a>.</p>
<p>Could you post your model definition as well as the input shapes so that we can reproduce this error, please?</p>

Karan_Chhabra: <p>Hi,<br>
I am trying to run the model in half-precision only. Sorry for the confusion.</p>
<p>My model seems to run for a few batches or even at times for an epoch but then it throughs error. The code of my model looks similar to the one below:</p>
<pre><code class="lang-auto">        images  = Variable(images).cuda()
        labels  = Variable(labels).cuda()


        net.half()  ## 
        images = images.half() ##

        #forward
        logits = net(images)
        logits = logits.float() ##

        loss = criterion(logits, labels) 

        loss.backward() 
        net.float() ##
        optimizer.step()
</code></pre>

ptrblck: <p><code>Variable</code>s are deprecated since PyTorch <code>0.4</code> so you can use tensors now.<br>
Could you post an executable code snippet using random tensors, so that we could reproduce the issue and debug further?</p>

Karan_Chhabra: <p>My training loop look like code below in actual, the code above was just an example, apologies for the confusion:</p>
<pre><code class="lang-auto">def train_classifier(classifier, train_loader, optimizer, criterion):
    classifier.half()
    classifier.train()
    loss = 0.0
    losses = []
    for i, (images, labels) in enumerate(train_loader):
        classifier.half()
        images, labels = images.to(device), labels.float().to(device)
        images = images.half()
        optimizer.zero_grad()
        logits = classifier(images)
        logits = logits.float()
        loss = criterion(logits, labels)
        loss = loss.float()
        loss.backward()
        classifier.float()
        optimizer.step()
        losses.append(loss)
    return torch.stack(losses).mean().item()
</code></pre>
<p>Any idea what could be wrong?</p>

ptrblck: <p>I guess the second error might be raised since you are converting the model to <code>half</code> and back to <code>float()</code> after again during the training, which could cause dtype mismatches.<br>
Could you explain your use case of converting the model back and forth and, if possible, post an executable code snippet as simple models (e.g. <code>resnet18</code>) seem to work?</p>

Karan_Chhabra: <p>Since this the first time I am trying to convert the model to half precision, so I just followed the post below. And it was converting the model to float and half, back and forth, so I thought this is the correct way.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37415">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37415" target="_blank" rel="noopener nofollow ugc">kaggle.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37415" target="_blank" rel="noopener nofollow ugc">Carvana Image Masking Challenge</a></h3>

  <p>Automatically identify the boundaries of the car in an image</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>But I am getting error even on the first epoch if I remove don’t convert back the model back to float. The modified code looks like:</p>
<pre><code class="lang-auto">def train_classifier(classifier, train_loader, optimizer, criterion):
    classifier.half()
    classifier.train()
    loss = 0.0
    losses = []
    for i, (images, labels) in enumerate(train_loader):
        images, labels = images.to(device), labels.float().to(device)
        images = images.half()
        optimizer.zero_grad()
        logits = classifier(images)
        logits = logits.float()
        loss = criterion(logits, labels)
        loss = loss.float()
        loss.backward()
        optimizer.step()
        losses.append(loss)
    return torch.stack(losses).mean().item()
</code></pre>
<p>The error which I am getting is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1caa532b332f514799038ceea140680cd852831.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1caa532b332f514799038ceea140680cd852831" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1caa532b332f514799038ceea140680cd852831.png" alt="image" data-base62-sha1="tVTXkh0AD8qSQyrzQlnan4F3Dod" width="690" height="159" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1405×324 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I would still recommend to use the automatic mixed-precision in case you want a stable FP16 training, where numerical sensitive operations are automatically performed in FP32.</p>
<aside class="quote no-group" data-username="Karan_Chhabra" data-post="7" data-topic="102622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karan_chhabra/48/30204_2.png" class="avatar"> Karan_Chhabra:</div>
<blockquote>
<p>The modified code looks like:</p>
</blockquote>
</aside>
<p>Could you still post the model definition and an executable code snippet to reproduce the issue, since I’m unable to run into this error using standard <code>torchvision</code> models.</p>

Karan_Chhabra: <p>I am using resnet34 as my base model, with last few layers as linear layer followed by sigmoid. My code looks like:</p>
<pre><code class="lang-auto">temp = nn.Sequential(
          nn.Dropout(p=0.5),
          nn.Linear(in_features=512, out_features=128),
          nn.ReLU(),
          nn.Linear(in_features=128, out_features=17, bias=True),
          nn.Sigmoid()
        )

classifier = torchvision.models.resnet34(pretrained=True)
classifier.fc = temp  
</code></pre>
<p>I am using Adam optimizer with BCELoss.<br>
And for every epoch I am just calling the above function</p>
<blockquote>
<p>train_classifier(classifier, )</p>
</blockquote>

ptrblck: <p>Thanks for the update.<br>
If I run your code snippet, I get invalid outputs after two iterations since the model is overflowing, which is creating an error in the criterion and thus a CUDA assert failure:</p>
<pre><code class="lang-python">/opt/conda/conda-bld/pytorch_1603729047590/work/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [0,0,0], thread: [33,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
</code></pre>
<p>Again, I would advice against using FP16 directly, over/underflows can easily happen.</p>

Karan_Chhabra: <p>Hi,</p>
<p>I am trying to run the autograded but facing the following issue:</p>
<p><code>AttributeError: module 'torch.cuda.amp' has no attribute 'autocast'</code></p>
<p>Any idea?</p>

ptrblck: <p>Could you update to the latest stable version (<code>.1.7.0</code>) and retry importing it?<br>
<code>torch.cuda.amp.autocast</code> was introduced in <code>1.6.0</code>, but I would recommend to use the latest version, since it ships with the latest bug fixes and additional features.</p>

Karan_Chhabra: <p>I am able to run the auto-cast by the loss is causing an issue, my code looks like:</p>
<pre><code class="lang-auto">with torch.cuda.amp.autocast():
          logits = classifier(images)
          loss = criterion(logits, labels)
</code></pre>
<p>And I am getting an error on loss calculation. The error mentions that I should use some other loss function other that BCELoss, but I need to have a sigmoid layer just before the output, so what kind of loss should I use as the pytorch is recommending me to use loss with logits.<br>
Or is there some way to make it work.</p>
<p>The error message is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88a50a51e1f8ef010ae5e60f134d9e013f932683.png" data-download-href="https://discuss.pytorch.org/uploads/default/88a50a51e1f8ef010ae5e60f134d9e013f932683" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a50a51e1f8ef010ae5e60f134d9e013f932683_2_690x152.png" alt="image" data-base62-sha1="juOx3GVlssMbh2MGYE36bpXtJTl" width="690" height="152" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a50a51e1f8ef010ae5e60f134d9e013f932683_2_690x152.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a50a51e1f8ef010ae5e60f134d9e013f932683_2_1035x228.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a50a51e1f8ef010ae5e60f134d9e013f932683_2_1380x304.png 2x" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1442×319 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your model should return the raw logits and you should use <code>nn.BCEWithLogitsLoss</code> as the criterion.<br>
If you want to see the probabilities, you could still apply <code>torch.sigmoid</code> to them, but don’t pass them to the loss function.</p>

Karan_Chhabra: <p>I am using the code below to convert the model into mixed precision. And I have also commented the sigmoid line but still I am facing the issue.</p>
<pre><code class="lang-auto">    for i, (images, labels) in enumerate(train_loader):
        images, labels = images.float().to(device), labels.float().to(device)
        optimizer.zero_grad()
        with torch.cuda.amp.autocast():
            logits = classifier(images)
            loss = criterion(logits, labels)
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        # Updates the scale for next iteration.
        scaler.update()
</code></pre>
<p>But I am getting the error on line <strong>scaler.scale(loss).backward()</strong>. I am getting the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2c03db6eab4ff402db20a71407868c039aa690b.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2c03db6eab4ff402db20a71407868c039aa690b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c03db6eab4ff402db20a71407868c039aa690b_2_690x159.png" alt="image" data-base62-sha1="u4o8TcnXlS3PJJ9m9mPe7xZmLiP" width="690" height="159" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c03db6eab4ff402db20a71407868c039aa690b_2_690x159.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c03db6eab4ff402db20a71407868c039aa690b_2_1035x238.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c03db6eab4ff402db20a71407868c039aa690b_2_1380x318.png 2x" data-dominant-color="F8F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1427×330 23.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Sorry for bugging you.</p>

ptrblck: <p>This code is working fine for me:</p>
<pre><code class="lang-python">temp = nn.Sequential(
          nn.Dropout(p=0.5),
          nn.Linear(in_features=512, out_features=128),
          nn.ReLU(),
          nn.Linear(in_features=128, out_features=17, bias=True),
        )

classifier = models.resnet34(pretrained=True)
classifier.fc = temp

device = 'cuda'  
classifier.to(device)
optimizer = torch.optim.SGD(classifier.parameters(), lr=1e-3)
scaler = torch.cuda.amp.GradScaler()

data = torch.randn(2, 3, 224, 224, device=device)
target = torch.randint(0, 2, (2, 17), device=device).float()
criterion = nn.BCEWithLogitsLoss()

for epoch in range(10):
    optimizer.zero_grad()
    with torch.cuda.amp.autocast():
        logits = classifier(data)
        loss = criterion(logits, target)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    # Updates the scale for next iteration.
    scaler.update()
    print('epoch {}, loss {:.3f}'.format(epoch, loss.item()))
</code></pre>

Karan_Chhabra: <p>Hi,</p>
<p>Seem like I was also manually calling half, due to which this error was occurring.<br>
Thank you</p>

harrycb: <p>During mixed precision training if the models weights aren’t fp16 then it’s not going to be fp16 in the end?</p>
<p>Or is it the case that you can convert to fp16 after training with AMP?</p>

parnyan_ataei: <p>I am trying to run a paper code with some changes, and I face a function which only work with float32, but may data are float16, I tried .half() and other suggestion about Mixed Precision, and face an error again, is there any solution?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8620d2354eb578142d20769bb1d69d4a09f77ea.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b8620d2354eb578142d20769bb1d69d4a09f77ea" title="function"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8620d2354eb578142d20769bb1d69d4a09f77ea_2_690x499.jpeg" alt="function" data-base62-sha1="qj7VzkUiPfjsUI0072E6oAPwy2S" width="690" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8620d2354eb578142d20769bb1d69d4a09f77ea_2_690x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8620d2354eb578142d20769bb1d69d4a09f77ea.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8620d2354eb578142d20769bb1d69d4a09f77ea.jpeg 2x" data-dominant-color="171819"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">function</span><span class="informations">904×654 58.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Disable <code>autocast</code> for this region if you are using amp as described in the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#autocast-and-custom-autograd-functions">docs</a>. If you are manually casting Inputs and parameters, transform the activation input used in this particular custom module to <code>float32</code>.</p>