'ConvolutionBackward0' Returned Nan Values in its 0th Output

NavinKumarMNK: <p>Logs :</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://wandb.ai/mnk/CrimeDetection/runs/4gtku1wy/files/output.log">
  <header class="source">
      <img src="https://cdn.wandb.ai/production/0ee2eca4d/favicon.png" class="site-icon" width="32" height="32">

      <a href="https://wandb.ai/mnk/CrimeDetection/runs/4gtku1wy/files/output.log" target="_blank" rel="noopener nofollow ugc">W&amp;B</a>
  </header>

  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/70720874?v=4" class="thumbnail onebox-avatar" width="460" height="460">

<h3><a href="https://wandb.ai/mnk/CrimeDetection/runs/4gtku1wy/files/output.log" target="_blank" rel="noopener nofollow ugc">mnk</a></h3>

  <p>Weights &amp; Biases, developer tools for machine learning</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am getting this error everytime in random training steps ranging from 300-3000</p>
<p>Pytorch-lightning Parameters:<br>
[AUTOENCODER_TRAIN]<br>
max_epochs = 500<br>
min_epochs = 250<br>
<span class="hashtag">#accelerator</span> = gpu<br>
benchmark = True<br>
weights_summary = full<br>
precision = 16<br>
gradient_clip_val = 5<br>
auto_lr_find = True<br>
auto_scale_batch_size = True<br>
auto_select_gpus = True<br>
check_val_every_n_epoch = 1<br>
fast_dev_run = False<br>
enable_progress_bar = True<br>
detect_anomaly=True<br>
accumulate_grad_batches=8<br>
track_grad_norm=2<br>
sync_batchnorm = True<br>
limit_val_batches=0.01<br>
limit_train_batches=0.01</p>
<p>DataLoader Code</p>
<pre><code class="lang-auto">'@Author:NavinKumarMNK'
# Add the parent directory to the path
import sys
import os
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))

import utils.utils as utils

# Import the required modules
import torch
import torch.nn as nn
import torch.nn.functional as F
import pytorch_lightning as pl
from torch.utils.data import Dataset, DataLoader, random_split
import cv2
import PIL
import numpy as np
from utils.preprocessing import ImagePreProcessing

class AutoEncoderDataset(Dataset):
    def __init__(self, batch_size:int, num_workers:int,
                    data_path, annotation_train) -&gt; None:
        super(AutoEncoderDataset, self).__init__()
        self.data_path = data_path
        self.annotation_train = open(annotation_train, 
                                        'r').read().splitlines()
        self.batch_size = int(batch_size)
        self.num_workers = int(num_workers)
        self.preprocessing = ImagePreProcessing()

        self.index = 0

    def __len__(self):
        return len(self.annotation_train)

    def __getitem__(self, index:int):
        
        i=0
        while True:
            i+=1
            if index+i &gt;= len(self.annotation_train):
                index = 0
            video_path = self.annotation_train[index+i]
            video_path = os.path.join(self.data_path, video_path) 
            
            cap = cv2.VideoCapture(video_path.strip())     
            count: int = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
            
            print(count, i)
            if not (cap.isOpened() and cap.get(cv2.CAP_PROP_FRAME_COUNT) &gt; 0):
                continue
            
            if count &lt; self.batch_size:
                ret_frames = np.random.randint(0, count, count)
            else:
                ret_frames= np.random.randint(0, count, self.batch_size)
            
                
            frames = []
            original = []
            # Get random frame indexes for batch size
            for frame in ret_frames:
                cap.set(1, frame)
                ret, frame = cap.read()
                if ret:
                    frame = np.transpose(frame, (2, 0, 1))
                    frame = self.preprocessing.transforms(torch.from_numpy(frame))
                    frame = self.preprocessing.preprocess(frame)
                    frame = self.preprocessing.augumentation(frame)
                    framex = self.preprocessing.improve(frame)
                    frames.append(framex)
                    framey = self.preprocessing.noise(frame)
                    original.append(framey)


            X = torch.stack(frames, dim=0)
            y = torch.stack(original, dim=0)
            
            if(torch.isnan(X).any() or torch.isnan(y).any()):
                print("reported")
                continue
            else:
                break

        return X, y    
    
class AutoEncoderDataModule(pl.LightningDataModule):
    def __init__(self, batch_size:int, num_workers:int,
                    data_path, annotation_train) -&gt; None:
        super(AutoEncoderDataModule, self).__init__()
        self.annotation_train = annotation_train
        self.batch_size = int(batch_size)
        self.num_workers = int(num_workers)
        self.data_path = data_path

    def setup(self, stage=None):
        full_dataset = AutoEncoderDataset(self.batch_size, self.num_workers,
                                           self.data_path, self.annotation_train)
        train_size = int(0.8 * len(full_dataset))
        val_size = int(0.1 * len(full_dataset))
        test_size = len(full_dataset) - train_size - val_size
        self.train_dataset, self.val_dataset, self.test_dataset = random_split(
            full_dataset, [train_size, val_size, test_size])
        
    def train_dataloader(self):
        return DataLoader(self.train_dataset, batch_size=1, num_workers=self.num_workers,
                           shuffle=True, drop_last=True, pin_memory=True)

    def val_dataloader(self):
        return DataLoader(self.train_dataset, batch_size=1, num_workers=self.num_workers,
                           shuffle=True, drop_last=True, pin_memory=True)

    def test_dataloader(self):
        return DataLoader(self.train_dataset, batch_size=1, num_workers=self.num_workers,
                           shuffle=True, drop_last=True, pin_memory=True)

if __name__ == '__main__':
    dataset_params = utils.config_parse('AUTOENCODER_DATASET')
    annotation_train = utils.dataset_image_autoencoder(
                            dataset_params['data_path'])
    dataset = AutoEncoderDataModule(**dataset_params, 
                    annotation_train=annotation_train)
    dataset.setup()
    
    train_loader = dataset.train_dataloader()
    from models.EfficientNetb3.AutoEncoder import AutoEncoder
    model = AutoEncoder().to('cuda:0').half()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    import time
    for i, (x, y) in enumerate(train_loader):
        x = x.view(x.size(1), x.size(2), x.size(3), x.size(4))
        y = y.view(y.size(1), y.size(2), y.size(3), y.size(4))
        out = model(x)
        print(out.shape)
    
        time.sleep(10)
        #train the model
        loss = F.mse_loss(out.to('cuda:0'), y.to('cuda:0'))
        loss.backward()
        
        #update the model
        optimizer.step()
        optimizer.zero_grad()


</code></pre>

ptrblck: <p>I don’t know what your training wrapper does and if Lightning is using e.g. mixed-precision training by default. If so, than note that invalid gradients are expected when <code>amp</code> is used with <code>float16</code> and the <code>GradScaler</code> will skip the parameter updates in this iteration before decreasing the scaling factor.<br>
In this case disable <code>torch.autograd.detect_anomaly</code>.<br>
If that’s not the case and <code>amp</code> is not used, check if you are seeing this issue using a specific input batch and make sure the model inputs contain valid values.</p>

NavinKumarMNK: <p>U mean that detecting anomaly causes this crash?<br>
yeah i’m using 16 precision training.<br>
Can u sugguest me which is best method to be followed in the training converting the model weights optimizers to float16 or only the data and doing calculation … i.2 O2 / O3.<br>
Which is best… i am doing a project training AutoEncoder on Surviellance images using Efficientb3 as encoder &amp; normal transposeConv as decoder. Actually even the model is not training after few steps… loss is stagnant, and when i convert the model in tensorrt i am recieving an eror that says weights are 16 prec but the input is float and the slow_cpu_conv of Efficient net is cannot able to process it… eventhough i am running in gpu</p>
<p>Actually this error i am getting in all the models i tried, including LSTM, CoAtNet etc., at one point of time i am recieiving the same error in any of the part. regarding LSTM, i am recieving it in<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7636ef2fa3583533a4537ea68307d222a43a2e60.png" data-download-href="https://discuss.pytorch.org/uploads/default/7636ef2fa3583533a4537ea68307d222a43a2e60" title="Screenshot from 2023-03-27 08-03-27"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7636ef2fa3583533a4537ea68307d222a43a2e60.png" alt="Screenshot from 2023-03-27 08-03-27" data-base62-sha1="gRM2l7oKiOIwp0HsgMtW1QIvs5O" width="690" height="258" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-03-27 08-03-27</span><span class="informations">1077×404 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Yes, invalid gradients are expected to occur during mixed-precision training in <code>float16</code> and anomaly detection will thus report false errors.<br>
Why do you enable it in the first place? This mode is only used during debugging.</p>

NavinKumarMNK: <p>I thought it would report the error before its happening and automatically solves it. Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/love_letter.png?v=12" title=":love_letter:" class="emoji" alt=":love_letter:" loading="lazy" width="20" height="20"> I will check and let it you know it works are not .<br>
So do u have any idea for training the Efficientb3 model as encoder for surveillance videos</p>
<p>And do you have any idea why am i getting weights in 16 bit and data in 32 bit and if i convert the data i am getting slow_cpu_error …</p>

ptrblck: <p>I don’t know why your parameters are in <code>float16</code>, but I would assume you are manually calling <code>.half()</code> on the model, which is not the recommended approach using <code>amp</code> as described in <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">the docs</a>.</p>

NavinKumarMNK: <p>Yeah but, the efficient-netb3 is a pytorch model not the pytorch-lightning model . Will it automatically sets their paramters type and device ?<br>
Yes, so setting amp back-end as ‘apex’ and with mode=o0, o1, o2, o3 has any effect extra effects, which will very useful?<br>
Then Slow_cpu_error is stating that when i use convert the pytorch model to onnx, its happening in the cpu. so 16bit is not supported in cpu calculations</p>

ptrblck: <p>PyTorch uses <code>float32</code> as the default <code>dtype</code> and will not apply mixed-precision training automatically for you.<br>
<code>apex.amp</code> is deprecated and shouldn’t be used anymore. Use the previously mentioned mixed-precision training via <code>torch.amp</code> instead.</p>

NavinKumarMNK: <p>Without using any .half() and doing the entire model training with precision = 16 in pytorch lightning Trainer.<br>
I am getting this error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ed3738f49af116c2e19e2b085414ecc174c0473.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ed3738f49af116c2e19e2b085414ecc174c0473" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0ed3738f49af116c2e19e2b085414ecc174c0473.png" alt="image" data-base62-sha1="279IsEK6d1ILfEwdD7Pdonzie43" width="690" height="311" data-dominant-color="15272F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1096×495 83.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I found out “deepspeed_stage_1” is causing this error.<br>
If i do training without pytorch lightning trainer() strategy is none i am able to start training and compiling it into onnx without any errors.<br>
And if the gpu is full i.e cuda out of memory… i gave in try statement if i get this cuda error then save the model in onnx format… at the time the same error occurs means… when we do computation in cpu i am getting this error</p>
<p>Any suggestions to solve it?</p>

ptrblck: <p><code>Lightning</code> still seems to call <code>.half()</code> on the model since the <code>weight</code> type is <code>torch.cuda.HalfFloat</code> as seen in the error message. I would still recommend to not use the deprecated <code>apex.amp</code> implementation and to avoid calling <code>half</code> on the model especially if you want to export the model to the CPU, which does not support all <code>float16</code> operations and will fail.</p>

NavinKumarMNK: <p>Yeah I dont use apex.amp, i am using pytorch_lightning’s “precision=16”. I dont get any error while doing this. But when i do with strategy set to deepspeed_stage_1 and distribute the workload, i am getting the same error<br>
And i am not exporting the model to cpu, it seems deep_speed is doing it.</p>

ryanxu8959: <blockquote>
<p>invalid gradients are expected to occur during mixed-precision training in <code>float16</code> and anomaly detection will thus report false errors</p>
</blockquote>
<p>I am curious how does lightning handle invalid gradients in this case?</p>

ptrblck: <p>I don’t know what exactly Lightning does for <code>precision=16</code>, but would assume it should be using the native <code>torch.amp</code> implementation.</p>