Transfer learning Tutorial - RuntimeError: CUDA error: device-side assert triggered

dnv: <p>Hello I am currently learning from the Transfer learning tutorial provided on the Pytorch webside.<br>
I  ran into this error while trying to train the model.<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
5 model_ft.fc = nn.Linear(num_ftrs, 2)<br>
6<br>
----&gt; 7 model_ft = model_ft.to(device)<br>
8<br>
9 criterion = nn.CrossEntropyLoss()</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/modules/module.py in to(self, *args, **kwargs)<br>
605             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)<br>
606<br>
 → 607         return self._apply(convert)<br>
608<br>
609     def register_backward_hook(</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)<br>
352     def _apply(self, fn):<br>
353         for module in self.children():<br>
 → 354             module._apply(fn)<br>
355<br>
356         def compute_should_use_set_data(tensor, tensor_applied):</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)<br>
374                 # <code>with torch.no_grad():</code><br>
375                 with torch.no_grad():<br>
 → 376                     param_applied = fn(param)<br>
377                 should_use_set_data = compute_should_use_set_data(param, param_applied)<br>
378                 if should_use_set_data:</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/modules/module.py in convert(t)<br>
603             if convert_to_format is not None and t.dim() == 4:<br>
604                 return t.to(device, dtype if t.is_floating_point() else None, non_blocking, memory_format=convert_to_format)<br>
 → 605             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)<br>
606<br>
607         return self._apply(convert)</p>
<p>RuntimeError: CUDA error: device-side assert triggered</p>
<h2>
<a name="it-is-clearly-an-indexing-error-i-tried-to-run-in-only-on-my-cpu-and-i-run-only-once-afterwards-was-coming-this-error-msg-1" class="anchor" href="#it-is-clearly-an-indexing-error-i-tried-to-run-in-only-on-my-cpu-and-i-run-only-once-afterwards-was-coming-this-error-msg-1"></a>It is clearly an indexing error. I tried to run in only on my cpu and i run only once. Afterwards was coming this error msg:</h2>
<p>IndexError                                Traceback (most recent call last)<br>
 in <br>
----&gt; 1 model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,<br>
2                        num_epochs=25)</p>
<p> in train_model(model, criterion, optimizer, scheduler, num_epochs)<br>
32                     outputs = model(inputs)<br>
33                     _, preds = torch.max(outputs, 1)<br>
—&gt; 34                     loss = criterion(outputs, labels)<br>
35<br>
36                     # backward + optimize only if in training phase</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
720             result = self._slow_forward(*input, **kwargs)<br>
721         else:<br>
 → 722             result = self.forward(*input, **kwargs)<br>
723         for hook in itertools.chain(<br>
724                 _global_forward_hooks.values(),</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/modules/loss.py in forward(self, input, target)<br>
945<br>
946     def forward(self, input: Tensor, target: Tensor) → Tensor:<br>
 → 947         return F.cross_entropy(input, target, weight=self.weight,<br>
948                                ignore_index=self.ignore_index, reduction=self.reduction)<br>
949</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)<br>
2420     if size_average is not None or reduce is not None:<br>
2421         reduction = _Reduction.legacy_get_string(size_average, reduce)<br>
 → 2422     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
2423<br>
2424</p>
<p>~/venvs/p_donov/lib/python3.8/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)<br>
2216                          .format(input.size(0), target.size(0)))<br>
2217     if dim == 2:<br>
 → 2218         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
2219     elif dim == 4:<br>
2220         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)</p>
<p>IndexError: Target 2 is out of bounds.</p>

ptrblck: <p>The raised error points towards an invalid index in the target:</p>
<pre><code class="lang-python">IndexError: Target 2 is out of bounds.
</code></pre>
<p>In a multi-class classification using <code>nn.CrossEntropyLoss</code> the model output is expected to have the shape <code>[batch_size, nb_classes]</code> while the target should have the shape <code>[batch_size]</code> and contain class indices in the range <code>[0, nb_classes-1]</code>.<br>
Since your target contains a class index of <code>2</code> this would mean that you are working with at least 3 different classes (class indices would be <code>[0, 1, 2]</code>) and your model output should thus have the shape <code>[batch_size, &gt;=3]</code>, which doesn’t seem to be the case.</p>

dnv: <p>I have another question regarding this example. Is the used network prototypical or matching network? While reading papers usually it is distinguished between matching and prototypical. It is also mentioned in the paper from (Snell et al., NIPS 2017) <a href="https://arxiv.org/abs/1703.05175" rel="noopener nofollow ugc">Prototypical Networks for Few-shot Learning</a>, that prototypical networks are better for training. Could somebody explain me how do I know with which kind of network am i dealing with?<br>
Thanks</p>

ptrblck: <p>I don’t know what the difference is and would thus recommend to create a new topic with this question.</p>