Torch.cuda.amp and Accuracy

bfeeny: <p>I converted my training loop to use AMP, and I notice my accuracy numbers are now all <code>0</code>.  What needs to be changed to work with AMP? <code>z</code> was calculated within a <code>with_autocast():</code> clause so I thought it should be fine, but apparently it is not.</p>
<pre><code class="lang-auto">        print("Training..............................\r", end='') 
        train_iter = iter(train_loader)
        next_batch = train_iter.next()
        next_batch = [_.cuda(non_blocking=True) for _ in next_batch ]
        for idx in range(len(train_loader)):
            image, meta, y=next_batch
            if(param['cache_on']):
                if(epoch == 0):
                    print(f"Loading Cache using train_loader {idx*train_loader.batch_size + image.shape[0]}\r", end='')   

            if idx + 1 != len(train_loader): 
                # start copying data of next batch
                next_batch = train_iter.next()
                next_batch = [ _.cuda(non_blocking=True) for _ in next_batch]


            optim.zero_grad()
            # AMP
            with autocast():
                z = model((image, meta))
                y_smo = y.float() * (1 - param['label_smoothing']) + 0.5 * param['label_smoothing']
                loss = criterion(z, y_smo.unsqueeze(1))

            # Before AMP
#             loss.backward()
            # AMP
            scaler.scale(loss).backward()


            # Before AMP
#             optim.step()
            # AMP
            scaler.step(optim)
            scaler.update()
            
            pred = torch.sigmoid(torch.round(z)) # round off sigmoid to obtain predictions  
            correct += (pred.cpu() == y.cpu().unsqueeze(1)).sum().item()  # tracking number of correctly predicted samples
            epoch_loss += loss.item()
        train_acc = correct / len(train_idx)
</code></pre>

ptrblck: <p>Do you suspect the accuracy calculation to be wrong or is the loss also not decreasing using amp?<br>
In the latter case, could you check the target calculation and see, if the expected values are calculated? If not, could you move the target calculation outside of the <code>autocast</code> region and recheck it?</p>
<p>Also, what kind of model are you using at the moment and I assume the training was working fine without amp?</p>

bfeeny: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Accuracy is fine if I remove the autocast clause.  Its just an EfficientNet B1 model.    The output of Accuracy is just 0, which is fine, I am not actually using that metric for this model, but obviously something has happened to z and perhaps there is a process to reverse it and so I can use it to further calculate my accuracy.  I can dig in and see where the number is being fudged, but I figure there may be something obvious I was doing wrong.</p>

ptrblck: <p>Which value are you using for the label smoothing and which criterion are you using?</p>
<p>Are you using a <code>sigmoid</code> at the end of your model and <code>nn.BCELoss</code>?<br>
If so, could you remove the sigmoid and use <code>nn.BCEWithLogitsLoss</code>?</p>
<p>The <code>pred</code> calculation looks a bit weird. If you are already using a <code>sigmoid</code> output,</p>
<pre><code class="lang-python">pred = torch.sigmoid(torch.round(z))
</code></pre>
<p>would create values in <code>[0.5, 0.73]</code>, wouldn’t it?</p>

bfeeny: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I am using BCE With Logits because regular BCE is not half-precision supported (I got an error to that effect when switching to half precision).  So I am using BCE w/ Logits and just tagging sigmoid on.  I suspected sigmoid, but I created a float16 tensor and ran it through sigmoid fine on GPU.  Sigmoid on CPU complains about half precision.  I am going to capture every value and find out what exactly is the issue.</p>
<p>I have since moved my label smoothing outside of the clause, so I modify y before the clause. My label smoothing is just basic, I am just modifying y in a basic way, there is nothing more to it than that.  For loss its BCE with Logits Loss.</p>

bfeeny: <p>How stupid of me!  When switching from BCE to BCEwL I mixed it up and I am doing sigmoid(round(x)) instead of round(sigmoid(x))!!!</p>