CUDA error, Kaggle dataset

Maryam_Darei: <p>I want to train a CNN model(ResNet50) in Pytorch with a dataset from Kaggle. I run my code on Kaggle GPU and I received a cuda error:</p>
<p>CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p>The number of classes are 15505, and for load the data I use a customized dataloader, not builtin function. This code previously worked with 10 classes on Colab with a pytorch dataloader.</p>
<p>Anybody can help?</p>

ptrblck: <p>Since you’ve changed the number of classes in your new workflow I would guess your target might contain invalid indices.<br>
Make sure the target contains class indices in <code>[0, nb_classes-1]</code> or rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the failing method in the stacktrace.</p>

Maryam_Darei: <p>I run this code with different number of classes many times, 2 classes, 10 classes, 870 classes. I did not receive this error. This time I changed the data loader. Is it possible the problem was because of this?</p>
<p>In addition, I did not understand this solution that you gave me: " rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the failing method in the stacktrace."</p>
<p>Thanks,</p>

Maryam_Darei: <p>You can check the code in my github:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/maryamdarei/herbarium2022/blob/main/herbarium2022_resnet50.ipynb">
  <header class="source">

      <a href="https://github.com/maryamdarei/herbarium2022/blob/main/herbarium2022_resnet50.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/maryamdarei/herbarium2022/blob/main/herbarium2022_resnet50.ipynb" target="_blank" rel="noopener nofollow ugc">maryamdarei/herbarium2022/blob/main/herbarium2022_resnet50.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "&lt;a href=\"https://colab.research.google.com/github/maryamdarei/herbarium2022/blob/main/herbarium2022_resnet50.ipynb\" target=\"_parent\"&gt;&lt;img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/&gt;&lt;/a&gt;"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SKQ4bH7qMGrA"
      },
      "source": [
        "# Making the Most of your Colab Subscription\n",
        "\n"
</code></pre>



  This file has been truncated. <a href="https://github.com/maryamdarei/herbarium2022/blob/main/herbarium2022_resnet50.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>If you cannot run the script in a terminal and need to run it in a notebook, execute it on the CPU to get a better stacktrace, which would then most likely point to an index error in <code>nn.CrossEntropyLoss</code>.</p>
<p>Also, thanks for the code. Could you double check this?</p>
<aside class="quote no-group" data-username="Maryam_Darei" data-post="1" data-topic="171274">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryam_darei/48/54395_2.png" class="avatar"> Maryam_Darei:</div>
<blockquote>
<p>The number of classes are 15505,</p>
</blockquote>
</aside>
<p>while you are using this in your code:</p>
<pre data-code-wrap="python"><code class="lang-python"># Define relevant variables for the ML task
num_classes = 15501
</code></pre>

Maryam_Darei: <p>I check with the cpu and I receive wired error, " loss is not definename ‘loss’ is not defined".<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93803a7427673bd07dc6453f28429c8fb3f19dcb.png" data-download-href="https://discuss.pytorch.org/uploads/default/93803a7427673bd07dc6453f28429c8fb3f19dcb" title="Screen Shot 2023-01-29 at 12.12.39 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93803a7427673bd07dc6453f28429c8fb3f19dcb_2_690x159.png" alt="Screen Shot 2023-01-29 at 12.12.39 PM" data-base62-sha1="l2QUNtqIzmQbZTd9O2YCC30GCP9" width="690" height="159" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93803a7427673bd07dc6453f28429c8fb3f19dcb_2_690x159.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93803a7427673bd07dc6453f28429c8fb3f19dcb_2_1035x238.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93803a7427673bd07dc6453f28429c8fb3f19dcb_2_1380x318.png 2x" data-dominant-color="F1D6D8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-01-29 at 12.12.39 PM</span><span class="informations">1394×322 43.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also I try with both,  num_classes = 15501 and num_classes = 15505. Because labels are originally numbered from 1 to 15505, but the competition data has 15501 labels because we lost four labels during data cleaning process. It is in json file.</p>

ptrblck: <p>The new issue points towards a script error and I guess you might be running into the exception:</p>
<pre><code class="lang-python">except Exception as e:
    log.error(f"Exception in data processing- skip and continue = {e}")
</code></pre>
<p>without skipping this iteration.<br>
Check if your <code>try/except</code> logic already raised a valid error and fix it first.</p>

Maryam_Darei: <p>I think I found the reason, but I dont know how I can solve it. It seems the model output is typical object classifier for the 1000 different classifications, not more. It might be because of the mismatch between the number of the classes and number of the last layer output. but I do not know how can I fix it! Do you have any solution?</p>

ptrblck: <p>Yes, this sounds reasonable as I didn’t realize you are using a standard ResNet without changing the output layer.<br>
Use:</p>
<pre><code class="lang-python">model = torchvision.models.resnet50()
model.fc = nn.Linear(2048, 15505)
model.to(device)
</code></pre>
<p>to replace the last linear layer with a new one returning logits for <code>15505</code> classes.</p>

Maryam_Darei: <p>Thanks, It seems it is ok now, but the data is enormous and need GPU to run.</p>