Cuda error: device side assert triggered at 56th epoch

Henry_Chibueze: <p>So my model was training and converging just fine for both training and testing cases and suddenly at the 56th epoch it threw a device-side assert triggered error.<br>
Here’s an image:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/482990fb43179449b4757e7d709bfe225c5daea7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/482990fb43179449b4757e7d709bfe225c5daea7" title="IMG-20220617-WA0057"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/482990fb43179449b4757e7d709bfe225c5daea7_2_588x500.jpeg" alt="IMG-20220617-WA0057" data-base62-sha1="ainqRJaSaSNK1fQImzSB5FZndnF" width="588" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/482990fb43179449b4757e7d709bfe225c5daea7_2_588x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/8/482990fb43179449b4757e7d709bfe225c5daea7.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/8/482990fb43179449b4757e7d709bfe225c5daea7.jpeg 2x" data-dominant-color="F4F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-20220617-WA0057</span><span class="informations">739×628 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cee6be482a5bc8141524edc401ffed3ebdbf0df.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5cee6be482a5bc8141524edc401ffed3ebdbf0df" title="IMG-20220617-WA0058"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cee6be482a5bc8141524edc401ffed3ebdbf0df_2_424x500.jpeg" alt="IMG-20220617-WA0058" data-base62-sha1="dg6K3GqFlRgayAqiglspxL0ugfd" width="424" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cee6be482a5bc8141524edc401ffed3ebdbf0df_2_424x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cee6be482a5bc8141524edc401ffed3ebdbf0df.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cee6be482a5bc8141524edc401ffed3ebdbf0df.jpeg 2x" data-dominant-color="F9DAD9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-20220617-WA0058</span><span class="informations">542×639 74.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m performing binary segmentation task.</p>
<p>I have checked and confirmed that my input values to the loss function are indeed within the range of (0, 1) as the final layer of my segmentation network has a sigmoid function.</p>
<p>I devised a loss function specific to my problem after some research, the loss function is a combination of BCEloss, hausdorff distance and dice loss like so:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15d9ec8b398d733e03a108b722b075d746f3c8dc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/15d9ec8b398d733e03a108b722b075d746f3c8dc" title="IMG_20220617_110239"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15d9ec8b398d733e03a108b722b075d746f3c8dc_2_312x500.jpeg" alt="IMG_20220617_110239" data-base62-sha1="37iVsYZqVSHV1qf6x3L382a7xYE" width="312" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15d9ec8b398d733e03a108b722b075d746f3c8dc_2_312x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15d9ec8b398d733e03a108b722b075d746f3c8dc_2_468x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15d9ec8b398d733e03a108b722b075d746f3c8dc_2_624x1000.jpeg 2x" data-dominant-color="14171C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20220617_110239</span><span class="informations">720×1153 219 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>both input and output of the model are float32 precision.</p>
<p>Initially I presumed that the model might be outputting NaN values, but then again if it did, the error would be pointing towards the loss computing function and not the batch_loss.backward() method.</p>
<p>I tried to reproduce this error again but with just a few 100 samples because it takes hours for this model to train with all samples and get to this point, but with the few 100 samples it ran fine and didn’t encounter any error.</p>
<p>Has anyone experienced this before?<br>
What could be the issue? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Henry_Chibueze: <p>After hours of debugging, I have finally discovered that the cause of this RuntimeError is one of the torchvision transforms (<code>transforms.RandomPerspective</code>). However, I still don’t know the reason why this transform gives NaN values.</p>
<p>Edit: it seems that the actual cause of the NaN values is the sequence of arrangement</p>
<pre><code class="lang-auto">class FirstChannelRandomInvert(object):
    def __init__(self, p):
        self.p = p
        self.invert_color = transforms.RandomInvert(p=self.p)

    def __call__(self, sample):
        #input shape: (..., 4, 224, 224)
        image, mask = sample[0].unsqueeze(dim=0), sample[1:]
        image = self.invert_color(image)
        sample = torch.cat((image, mask), dim=0)
        return sample


class FirstChannelRandomGaussianBlur(object):
    def __init__(self, p, kernel_size=(5, 9), sigma=(0.1, 5)):
        self.p = p
        self.kernel_size = kernel_size
        self.sigma = sigma
        self.gaussian_blur = transforms.GaussianBlur(kernel_size=self.kernel_size, sigma=self.sigma)

    def __call__(self, sample):
        #input shape: (..., 4, 224, 224)
        image, mask = sample[0].unsqueeze(dim=0), sample[1:]
        randn = np.random.rand()
        if randn &lt; self.p: image = self.gaussian_blur(image)
        sample = torch.cat((image, mask), dim=0)
        return sample


data_transforms = transforms.Compose([
  transforms.RandomHorizontalFlip(p=0.5),
  transforms.RandomVerticalFlip(p=0.5),
  FirstChannelRandomInvert(p=0.5),
  FirstChannelRandomGaussianBlur(p=0.5),
  transforms.RandomRotation((0, 360)),
  transforms.RandomPerspective(distortion_scale=0.6, p=0.5)
])
</code></pre>
<p>Although, I see nothing wrong in this, but when I change the sequence a bit, it stops producing NaN values.</p>

ptrblck: <aside class="quote no-group" data-username="Henry_Chibueze" data-post="2" data-topic="154387">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_chibueze/48/28458_2.png" class="avatar"> Henry_Chibueze:</div>
<blockquote>
<p>it seems that the actual cause of the NaN values is the sequence of arrangement</p>
</blockquote>
</aside>
<p>Does this mean that the NaN values are not created anymore if you swap the order of transformations or what is the “sequence” referring to?</p>

Henry_Chibueze: <p>Good day.<br>
Yes if I swap the order of the transform, the NaN values are not created anymore</p>