RuntimeError: size mismatch (got input: [6422528], target: [802816])

hedeya1980: <p>I’m trying to adjust a binary segmentation U-net model, to be able to train a multi-class U-net on the German Asfalt Pavement Distress (GAPs) dataset.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 263, in &lt;module&gt;
    train(train_loader, model, criterion, optimizer, validate, args)
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 124, in train
    loss = criterion(masks_probs_flat, true_masks_flat)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py", line 1165, in forward
    label_smoothing=self.label_smoothing)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2996, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: size mismatch (got input: [6422528], target: [802816])
</code></pre>
<p>The code files, and the dataset are available through the following link:<br>
<a href="https://drive.google.com/drive/folders/14NQdtMXokIixBJ5XizexVECn23Jh9aTM?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/14NQdtMXokIixBJ5XizexVECn23Jh9aTM?usp=sharing</a></p>
<p>The following link is for the last stackoverflow question (before I change the criterion to use nn.CrossEntropyLoss). I’m totally new to pytorch, and I look forward to receiving your valuable advice.</p>
<p>The following is the code in “train_unet_GAPs.py”:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from unet.unet_transfer import UNet16, UNetResNet
from pathlib import Path
import torchvision.transforms as transforms
from torch.utils.data import DataLoader, Dataset, random_split
import torch.nn.functional as F
from torch.autograd import Variable
import shutil
from data_loader import ImgDataSet
import os
import argparse
import tqdm
import numpy as np
import scipy.ndimage as ndimage

class AverageMeter(object):
    """Computes and stores the average and current value"""
    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

def create_model(device, type ='vgg16'):
    if type == 'vgg16':
        print('create vgg16 model')
        model = UNet16(pretrained=True)
    elif type == 'resnet101':
        encoder_depth = 101
        num_classes = 8
        print('create resnet101 model')
        model = UNetResNet(encoder_depth=encoder_depth, num_classes=num_classes, pretrained=True)
    elif type == 'resnet34':
        encoder_depth = 34
        num_classes = 8
        print('create resnet34 model')
        model = UNetResNet(encoder_depth=encoder_depth, num_classes=num_classes, pretrained=True)
    else:
        assert False
    model.eval()
    return model.to(device)

def adjust_learning_rate(optimizer, epoch, lr):
    """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
    lr = lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

def find_latest_model_path(dir):
    model_paths = []
    epochs = []
    for path in Path(dir).glob('*.pt'):
        if 'epoch' not in path.stem:
            continue
        model_paths.append(path)
        parts = path.stem.split('_')
        epoch = int(parts[-1])
        epochs.append(epoch)

    if len(epochs) &gt; 0:
        epochs = np.array(epochs)
        max_idx = np.argmax(epochs)
        return model_paths[max_idx]
    else:
        return None

def train(train_loader, model, criterion, optimizer, validation, args):

    latest_model_path = find_latest_model_path(args.model_dir)

    best_model_path = os.path.join(*[args.model_dir, 'model_best.pt'])

    if latest_model_path is not None:
        state = torch.load(latest_model_path)
        epoch = state['epoch']
        model.load_state_dict(state['model'])
        epoch = epoch

        #if latest model path does exist, best_model_path should exists as well
        assert Path(best_model_path).exists() == True, f'best model path {best_model_path} does not exist'
        #load the min loss so far
        best_state = torch.load(latest_model_path)
        min_val_los = best_state['valid_loss']

        print(f'Restored model at epoch {epoch}. Min validation loss so far is : {min_val_los}')
        epoch += 1
        print(f'Started training model from epoch {epoch}')
    else:
        print('Started training model from epoch 0')
        epoch = 0
        min_val_los = 9999

    valid_losses = []
    for epoch in range(epoch, args.n_epoch + 1):

        adjust_learning_rate(optimizer, epoch, args.lr)

        tq = tqdm.tqdm(total=(len(train_loader) * args.batch_size))
        tq.set_description(f'Epoch {epoch}')

        losses = AverageMeter()

        model.train()
        for i, (input, target) in enumerate(train_loader):
            input_var  = Variable(input).cuda()
            target_var = Variable(target).cuda()

            masks_pred = model(input_var)

            masks_probs_flat = masks_pred.view(-1)
            true_masks_flat  = target_var.view(-1)
            print(masks_probs_flat.shape, true_masks_flat.shape)

            loss = criterion(masks_probs_flat, true_masks_flat)
            losses.update(loss)
            tq.set_postfix(loss='{:.5f}'.format(losses.avg))
            tq.update(args.batch_size)

            # compute gradient and do SGD step
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        valid_metrics = validation(model, valid_loader, criterion)
        valid_loss = valid_metrics['valid_loss']
        valid_losses.append(valid_loss)
        print(f'\tvalid_loss = {valid_loss:.5f}')
        tq.close()

        #save the model of the current epoch
        epoch_model_path = os.path.join(*[args.model_dir, f'model_epoch_{epoch}.pt'])
        torch.save({
            'model': model.state_dict(),
            'epoch': epoch,
            'valid_loss': valid_loss,
            'train_loss': losses.avg
        }, epoch_model_path)

        if valid_loss &lt; min_val_los:
            min_val_los = valid_loss

            torch.save({
                'model': model.state_dict(),
                'epoch': epoch,
                'valid_loss': valid_loss,
                'train_loss': losses.avg
            }, best_model_path)

def validate(model, val_loader, criterion):
    losses = AverageMeter()
    model.eval()
    with torch.no_grad():

        for i, (input, target) in enumerate(val_loader):
            input_var = Variable(input).cuda()
            target_var = Variable(target).long().cuda()
            

            output = model(input_var)
            loss = criterion(output, target_var)

            losses.update(loss.item(), input_var.size(0))

    return {'valid_loss': losses.avg}

def save_check_point(state, is_best, file_name = 'checkpoint.pth.tar'):
    torch.save(state, file_name)
    if is_best:
        shutil.copy(file_name, 'model_best.pth.tar')

def calc_crack_pixel_weight(mask_dir):
    avg_w = 0.0
    n_files = 0
    for path in Path(mask_dir).glob('*.*'):
        n_files += 1
        m = ndimage.imread(path)
        ncrack = np.sum((m &gt; 0)[:])
        w = float(ncrack)/(m.shape[0]*m.shape[1])
        avg_w = avg_w + (1-w)

    avg_w /= float(n_files)

    return avg_w / (1.0 - avg_w)

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='PyTorch ImageNet Training')
    parser.add_argument('-n_epoch', default=10, type=int, metavar='N', help='number of total epochs to run')
    parser.add_argument('-lr', default=0.001, type=float, metavar='LR', help='initial learning rate')
    parser.add_argument('-momentum', default=0.9, type=float, metavar='M', help='momentum')
    parser.add_argument('-print_freq', default=20, type=int, metavar='N', help='print frequency (default: 10)')
    parser.add_argument('-weight_decay', default=1e-4, type=float, metavar='W', help='weight decay (default: 1e-4)')
    parser.add_argument('-batch_size',  default=4, type=int,  help='weight decay (default: 1e-4)')
    #parser.add_argument('-batch_size',  default=2, type=int,  help='weight decay (default: 1e-4)')
    #parser.add_argument('-num_workers', default=4, type=int, help='output dataset directory')
    parser.add_argument('-num_workers', default=2, type=int, help='output dataset directory')

    parser.add_argument('-data_dir',type=str, help='input dataset directory')
    parser.add_argument('-model_dir', type=str, help='output dataset directory')
    parser.add_argument('-model_type', type=str, required=False, default='resnet101', choices=['vgg16', 'resnet101', 'resnet34'])

    args = parser.parse_args()
    os.makedirs(args.model_dir, exist_ok=True)

    DIR_IMG  = os.path.join(args.data_dir, 'images')
    DIR_MASK = os.path.join(args.data_dir, 'masks')

    img_names  = [path.name for path in Path(DIR_IMG).glob('*.jpg')]
    mask_names = [path.name for path in Path(DIR_MASK).glob('*.png')]

    print(f'total images = {len(img_names)}')

    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

    model = create_model(device, args.model_type)

    optimizer = torch.optim.SGD(model.parameters(), args.lr,
                                momentum=args.momentum,
                                weight_decay=args.weight_decay)

    #crack_weight = 0.4*calc_crack_pixel_weight(DIR_MASK)
    #print(f'positive weight: {crack_weight}')
    #criterion = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor([crack_weight]).to('cuda'))
    #criterion = nn.BCEWithLogitsLoss().to('cuda')
    criterion = nn.CrossEntropyLoss().to('cuda')

    #channel_means = [0.485, 0.456, 0.406]
    #channel_stds  = [0.229, 0.224, 0.225]
    channel_means = [0.5]
    channel_stds  = [0.5]
    train_tfms = transforms.Compose([transforms.Resize((448,448)), transforms.ToTensor(),
                                     transforms.Normalize(channel_means, channel_stds)])

    val_tfms = transforms.Compose([transforms.Resize((448,448)), transforms.ToTensor(),
                                   transforms.Normalize(channel_means, channel_stds)])

    mask_tfms = transforms.Compose([transforms.Resize((448,448)), transforms.ToTensor(),transforms.Normalize((0.5,), (0.5,))])
    '''
    mask_tfms = transforms.Compose([
    transforms.ToTensor(),
    transforms.Lambda(lambda x: x.repeat(3,1,1)),
    transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))])
    '''
    dataset = ImgDataSet(img_dir=DIR_IMG, img_fnames=img_names, img_transform=train_tfms, mask_dir=DIR_MASK, mask_fnames=mask_names, mask_transform=mask_tfms)
    train_size = int(0.85*len(dataset))
    valid_size = len(dataset) - train_size
    train_dataset, valid_dataset = random_split(dataset, [train_size, valid_size])

    train_loader = DataLoader(train_dataset, args.batch_size, shuffle=False, pin_memory=torch.cuda.is_available(), num_workers=args.num_workers)
    valid_loader = DataLoader(valid_dataset, args.batch_size, shuffle=False, pin_memory=torch.cuda.is_available(), num_workers=args.num_workers)

    #model.cuda()
    model.to(torch.device("cuda:0"))

    train(train_loader, model, criterion, optimizer, validate, args)

</code></pre>
<p>The following is the code in “data_loader.py”:</p>
<pre><code class="lang-auto">import os
import numpy as np
from torch.utils.data import DataLoader, Dataset
import random
from PIL import Image
import matplotlib.pyplot as plt

class ImgDataSet(Dataset):
    def __init__(self, img_dir, img_fnames, img_transform, mask_dir, mask_fnames, mask_transform):
        self.img_dir = img_dir
        self.img_fnames = img_fnames
        self.img_transform = img_transform

        self.mask_dir = mask_dir
        self.mask_fnames = mask_fnames
        self.mask_transform = mask_transform

        self.seed = np.random.randint(2147483647)

    def __getitem__(self, i):
        fname = self.img_fnames[i]
        fpath = os.path.join(self.img_dir, fname)
        img = Image.open(fpath)
        if self.img_transform is not None:
            random.seed(self.seed)
            img = self.img_transform(img)
            #print('image shape', img.shape)

        mname = self.mask_fnames[i]
        mpath = os.path.join(self.mask_dir, mname)
        mask = Image.open(mpath)
        #print('khanh1', np.min(test[:]), np.max(test[:]))
        if self.mask_transform is not None:
            mask = self.mask_transform(mask)
            #print('mask shape', mask.shape)
            #print('khanh2', np.min(test[:]), np.max(test[:]))

        return img, mask #torch.from_numpy(np.array(mask, dtype=np.int64))

    def __len__(self):
        return len(self.img_fnames)


class ImgDataSetJoint(Dataset):
    def __init__(self, img_dir, img_fnames, joint_transform, mask_dir, mask_fnames, img_transform = None, mask_transform = None):
        self.joint_transform = joint_transform

        self.img_dir = img_dir
        self.img_fnames = img_fnames
        self.img_transform = img_transform

        self.mask_dir = mask_dir
        self.mask_fnames = mask_fnames
        self.mask_transform = mask_transform

        self.seed = np.random.randint(2147483647)

    def __getitem__(self, i):
        fname = self.img_fnames[i]
        fpath = os.path.join(self.img_dir, fname)
        img = Image.open(fpath)

        mname = self.mask_fnames[i]
        mpath = os.path.join(self.mask_dir, mname)
        mask = Image.open(mpath)

        if self.joint_transform is not None:
            img, mask = self.joint_transform([img, mask])

        #debug
        # img = np.asarray(img)
        # mask = np.asarray(mask)
        # plt.subplot(121)
        # plt.imshow(img)
        # plt.subplot(122)
        # plt.imshow(img)
        # plt.imshow(mask, alpha=0.4)
        # plt.show()

        if self.img_transform is not None:
            img = self.img_transform(img)

        if self.mask_transform is not None:
            mask = self.mask_transform(mask)

        return img, mask #torch.from_numpy(np.array(mask, dtype=np.int64))

    def __len__(self):
        return len(self.img_fnames)
</code></pre>

KFrank: <p>Hi Mohamed!</p>
<aside class="quote no-group" data-username="hedeya1980" data-post="1" data-topic="154254" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ccd318/48.png" class="avatar"> hedeya1980:</div>
<blockquote>
<p>I’m trying to adjust a binary segmentation U-net model, to be able to train a multi-class U-net</p>
</blockquote>
</aside>
<p>The short story is that you are using <code>CrossEntropyLoss</code> incorrectly.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">RuntimeError: size mismatch (got input: [6422528], target: [802816])
</code></pre>
<pre><code class="lang-auto">    elif type == 'resnet101':
        encoder_depth = 101
        num_classes = 8
        print('create resnet101 model')
        model = UNetResNet(encoder_depth=encoder_depth, num_classes=num_classes, pretrained=True)
...
        for i, (input, target) in enumerate(train_loader):
            input_var  = Variable(input).cuda()
            target_var = Variable(target).cuda()

            masks_pred = model(input_var)

            masks_probs_flat = masks_pred.view(-1)
            true_masks_flat  = target_var.view(-1)
            print(masks_probs_flat.shape, true_masks_flat.shape)

            loss = criterion(masks_probs_flat, true_masks_flat)
</code></pre>
</blockquote>
</aside>
<p>Making some guesses about your use case, I would expect both<br>
<code>input_var</code> and <code>target_var</code> to have shape<br>
<code>[nBatch = 4, nChannels = 1, 448, 448]</code>.  (You might not have<br>
an <code>nChannels</code> dimension, and the values of <code>nBatch</code> and <code>nChannels</code><br>
aren’t really relevant to your issue.</p>
<p>However, your multi-class model will add an <code>nClass = 8</code> dimension to<br>
its predictions.</p>
<p>Therefore I would expect <code>masks_pred</code> to have shape<br>
<code>[nBatch = 4, nChannels = 1, 448, 448, nClass = 8]</code></p>
<p>At this point <code>masks_pred</code> and <code>target_var</code> are appropriate to pass into<br>
<code>CrossEntropyLoss</code> (with the proviso that <code>target_var</code> should consist of<br>
<code>long</code> class labels that run from <code>0</code> to <code>nClass - 1  = 7</code>).</p>
<p>Your mistake is that you then <code>.view()</code> <code>masks_prod_flat</code> and<br>
<code>true_masks_flat</code> as one-dimensional tensors.  (When you print<br>
out their shapes you presumably get <code>[6422528]</code> and <code>[802816]</code>,<br>
respectively.)  This just isn’t right for <code>CrossEntropyLoss</code>, hence<br>
the error.</p>
<p>Best.</p>
<p>K. Frank</p>

hedeya1980: <p>Hi K. Frank,</p>
<p>Thanks for your reply. Could you pls help to advise what should I revise in the code to get it running. If I commented out the .view() lines, and use <code>loss = criterion(masks_pred, target_var)</code> I got the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 265, in &lt;module&gt;
    train(train_loader, model, criterion, optimizer, validate, args)
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 125, in train
    loss = criterion(masks_pred, target_var)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py", line 1165, in forward
    label_smoothing=self.label_smoothing)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2996, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of size: : [4, 1, 448, 448]
</code></pre>

KFrank: <p>Hi Mohamed!</p>
<aside class="quote no-group quote-modified" data-username="hedeya1980" data-post="3" data-topic="154254" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ccd318/48.png" class="avatar"> hedeya1980:</div>
<blockquote>
<p>use <code>loss = criterion(masks_pred, target_var)</code> I got the following error:</p>
<pre><code class="lang-auto">RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of size: : [4, 1, 448, 448]
</code></pre>
</blockquote>
</aside>
<p>That’s my fault.  I was forgetting that <code>CrossEntropyLoss</code> expects its <code>input</code><br>
(your <code>masks_pred</code>) to have its second dimension be its <code>nClass</code> dimension.</p>
<p>So if <code>target_var</code> has shape <code>[4, 1, 448, 448]</code> (and <code>nClass = 8</code>), you<br>
would want <code>masks_pred</code> to have shape <code>[4, 8, 1, 448, 448]</code>.  If your<br>
model produces a <code>masks_pred</code> with shape <code>[4, 1, 448, 448, 8]</code> (i.e.,<br>
with the <code>nClass</code> dimension last), you could swap the dimensions around<br>
by calling your loss criterion as:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = criterion (masks_pred.permute (0, 4, 1, 2, 3), target_var)
</code></pre>
<p>(If this doesn’t work, first try <code>squeeze()</code>ing away the singleton channels<br>
dimensions, and if it still doesn’t work, print out the shapes of <code>masks_pred</code><br>
and <code>target_var</code> and tell us what version of pytorch you are using so we can<br>
be sure what we’re looking at.)</p>
<p>Best.</p>
<p>K. Frank</p>

hedeya1980: <p>Hi K. Frank,<br>
The shape of masks_pred is [4, 8, 448, 448] not [4, 8, 1, 448, 448].<br>
The shape of target_var is [4, 1, 448, 448].</p>
<p>torch version is 1.11.0+cu113.</p>
<p>Best regards,<br>
Mohamed</p>

KFrank: <p>Hi Mohamed!</p>
<aside class="quote no-group" data-username="hedeya1980" data-post="5" data-topic="154254" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ccd318/48.png" class="avatar"> hedeya1980:</div>
<blockquote>
<p>The shape of masks_pred is [4, 8, 448, 448] not [4, 8, 1, 448, 448].<br>
The shape of target_var is [4, 1, 448, 448].</p>
</blockquote>
</aside>
<p>That clears things up.</p>
<p><code>masks_pred</code> correctly has its <code>nChannels</code> dimension as its second<br>
dimension.  But <code>target_var</code> has that extra singleton dimension that<br>
doesn’t line up with anything in <code>masks_pred</code>.  (In the most common<br>
integer categorical class labels use case, <code>masks_pred</code> should have<br>
one more dimension – the <code>nChannels</code> dimension – than <code>target_var</code>.)</p>
<p>Just <code>squeeze()</code> that singleton dimension away.  Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.11.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; masks_pred = torch.randn (4, 8, 448, 448)
&gt;&gt;&gt; target_var = torch.randint (8, (4, 1, 448,  448))
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (masks_pred, target_var.squeeze())
tensor(2.4896)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

hedeya1980: <p>Thanks K. Frank.<br>
We are getting closer. The dimension errors are gone after I revised the loss as you suggested <code>loss = criterion(masks_pred, target_var.squeeze())</code><br>
Now, I got the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 265, in &lt;module&gt;
    train(train_loader, model, criterion, optimizer, validate, args)
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 125, in train
    loss = criterion(masks_pred, target_var.squeeze())
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py", line 1165, in forward
    label_smoothing=self.label_smoothing)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2996, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: expected scalar type Long but found Float
</code></pre>
<p>To try to correct the above error, I revised the <code>target_var </code> to be as follows:</p>
<pre><code class="lang-auto">target_var = Variable(target).type(torch.LongTensor).cuda()
</code></pre>
<p>However, I got the following error:</p>
<pre><code class="lang-auto">../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [977,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [978,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [979,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [980,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [981,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [982,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [983,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [984,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [985,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [986,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [987,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [988,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [989,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [990,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [991,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [864,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [865,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [866,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [867,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [868,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [869,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [870,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [871,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [872,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [873,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [874,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [875,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [876,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [877,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [878,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [879,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [880,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [881,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [882,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [883,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [884,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [885,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [992,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [993,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [994,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [995,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [996,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [997,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [998,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [999,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1000,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1001,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1002,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1003,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1004,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1005,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1006,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1007,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1008,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1009,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1010,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1011,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1012,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [1013,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [657,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [658,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [659,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [660,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [661,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [662,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [663,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [664,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [665,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [666,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [667,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [668,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [669,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [670,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [671,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [800,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [801,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [802,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [803,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [804,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [805,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [806,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [807,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [808,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [809,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [810,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [811,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [812,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [813,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [814,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [815,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [816,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [817,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [818,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [819,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [820,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [821,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [288,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [289,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [290,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [291,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [292,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [293,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [294,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [295,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [296,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [297,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [298,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [299,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [300,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [301,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [302,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [303,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [304,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [305,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [306,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [307,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [308,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [309,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [465,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [466,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [467,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [468,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [469,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [470,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [471,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [472,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [473,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [474,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [475,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [476,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [477,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [478,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:93: nll_loss2d_forward_kernel: block: [3,0,0], thread: [479,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
Traceback (most recent call last):
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 265, in &lt;module&gt;
    train(train_loader, model, criterion, optimizer, validate, args)
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 127, in train
    tq.set_postfix(loss='{:.5f}'.format(losses.avg))
  File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 627, in __format__
    return self.item().__format__(format_spec)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>I look forward to getting your advice.</p>

hedeya1980: <p>When I printed <code>print(mask.min(), mask.max())</code>, I got the following result:</p>
<pre><code class="lang-auto">tensor(-1.) tensor(-0.9529)
tensor(-0.9922) tensor(-0.9529)
tensor(-1.) tensor(-0.9529)
tensor(-1.) tensor(-0.9451)
tensor(-0.9922) tensor(-0.9451)
tensor(-0.9922) tensor(-0.9451)
tensor(-1.) tensor(-0.9529)
</code></pre>
<p>I can’t understand why is that. The masks have are grayscale images with the following coding:</p>
<pre><code class="lang-auto"># Mask images are stored as grey value images.
# Coding of the grey values:
#    0 = VOID
#    1 = intact road,
#    2 = applied patch,
#    3 = pothole,
#    4 = inlaid patch,
#    5 = open joint,
#    6 = crack
#    7 = street inventory
</code></pre>

KFrank: <p>Hi Mohamed!</p>
<aside class="quote no-group quote-modified" data-username="hedeya1980" data-post="7" data-topic="154254" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ccd318/48.png" class="avatar"> hedeya1980:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: expected scalar type Long but found Float
</code></pre>
<p>To try to correct the above error, I revised the <code>target_var </code> to be as follows:</p>
<pre><code class="lang-auto">target_var = Variable(target).type(torch.LongTensor).cuda()
</code></pre>
<p>However, I got the following error:</p>
<pre><code class="lang-auto">block: [3,0,0], thread: [977,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
</code></pre>
</blockquote>
</aside>
<p>First, some advice:</p>
<p>You can’t just modify one bit of code in order to apply to some rather<br>
different use case and hope it works.  You will almost always have to<br>
modify other bits of your code to make them consistent with your<br>
initial modification.</p>
<p>You are learning, detail by detail, that <code>CrossEntropyLoss</code> takes a<br>
<code>target</code> that has a different form than the one used by whatever loss<br>
criterion you were using prior to your modification.</p>
<p>Quoting from my earlier post:</p>
<blockquote>
<p>At this point <code>masks_pred</code> and <code>target_var</code> are appropriate to pass into<br>
<code>CrossEntropyLoss</code> (with the proviso that <code>target_var</code> should consist of<br>
<code>long</code> class labels that run from <code>0</code> to <code>nClass - 1  = 7</code>).</p>
</blockquote>
<p>Your errors are:  First, <code>target</code> has to be of type <code>long</code>; and second, its values<br>
have to range from <code>0</code> to <code>7</code>.</p>
<p>Please take a look at the documentation for <a href="https://pytorch.org/docs/1.11/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss</a> to see how<br>
it is designed to be used, and verify that you are using it correctly.</p>
<p>(As an aside, <code>Variable</code> has been deprecated for quite some time<br>
now and doesn’t do anything.  Just use tensors, possibly setting<br>
<code>requires_grad = True</code>, but only when appropriate.)</p>
<p>Best.</p>
<p>K. Frank</p>

KFrank: <p>Hi Mohamed!</p>
<aside class="quote no-group quote-modified" data-username="hedeya1980" data-post="8" data-topic="154254" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ccd318/48.png" class="avatar"> hedeya1980:</div>
<blockquote>
<p>When I printed <code>print(mask.min(), mask.max())</code>, I got the following result:</p>
<pre><code class="lang-auto">tensor(-1.) tensor(-0.9529)
...
</code></pre>
<p>I can’t understand why is that. The masks have are grayscale images with the following coding:</p>
<pre><code class="lang-auto"># Mask images are stored as grey value images.
# Coding of the grey values:
#    0 = VOID
#    1 = intact road,
...
</code></pre>
</blockquote>
</aside>
<p>You say that you have masks on disk that have pixel values that run from<br>
<code>0</code> through <code>7</code>.  This sounds just right for your use case.  (You might double<br>
check this, though.)</p>
<p>But when you print out values for <code>mask</code> in memory, you get floating-point<br>
values near <code>-1.0</code> – a problem for your use case.</p>
<p>You need to step through the various layers of processing that read your<br>
mask from disk and turn it into the tensor <code>mask</code>.  Something is changing<br>
those values somewhere along the line.</p>
<p>Best.</p>
<p>K. Frank</p>

hedeya1980: <p>Hi K. Frank,</p>
<p>When I changed the line <code>loss = criterion(masks_probs_flat, true_masks_flat)</code> into <code>loss = criterion(masks_pred, target_var.squeeze())</code>, the training worked.<br>
However, I think there is still something wrong.<br>
The results of <code>print(mask.min(), mask.max())</code> are now positive floats such as <code>print(mask.min(), mask.max())</code></p>
<p>I think what change the values from 0 through 7 into floats are the following transforms (I think the resizing is what changes the mask values):</p>
<pre><code class="lang-auto">train_tfms = transforms.Compose([transforms.Resize((448,448)), transforms.ToTensor()])
val_tfms = transforms.Compose([transforms.Resize((448,448)), transforms.ToTensor()])
mask_tfms = transforms.Compose([transforms.Resize((448,448)), transforms.ToTensor()])
</code></pre>
<p>I had to add the resizing because I understood from a previous discussion that Resnet only accepts square images as input.</p>
<p>I appreciate all your help so far. I look forward to receiving further advice from you.</p>