Error training ResNet 50 model

Naina_K: <p>Hi everyone! When I try to run my code below, I get a runtime error saying: Given groups=1, weight of size [64, 3, 7, 7], expected input[32, 256, 256, 3] to have 3 channels, but got 256 channels instead.</p>
<p>However, I wrote code to account for this issue by creating a custom dataset, but I’m not sure what’s wrong.</p>
<p><strong>Custom Dataset</strong></p>
<pre><code class="lang-auto">class BreastCancerImages(Dataset):

    def __init__(self, csv_file, root_dir, n_channel=3, transform=None, lbl_col_idx=6, image_file_dir='/kaggle/input/rsna-mammography-images-as-pngs/images_as_pngs/train_images_processed'):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied on a sample.
        """
        self.n_channel = n_channel
        self.d_train = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform
        self.lbl_col_idx = lbl_col_idx
        self.files = list(glob(os.path.join(image_file_dir, "**", "*.png")))
    def __len__(self):
        return len(self.d_train)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        
        # x
        img_name = self.files[idx]
        image = open_img(img_name, self.n_channel)
        image = torch.tensor(image)
        
        # y
        labels = self.d_train.iloc[idx, self.lbl_col_idx]
        labels = np.array([labels])
        labels = torch.tensor(labels.astype('int'))
        return image, labels
</code></pre>
<p><strong>ResNet 50</strong></p>
<pre><code class="lang-auto">resnet50 = models.resnet50(pretrained = True)
resnet50

for param in resnet50.parameters():
    param.requires_grad = False
    
resnet50.fc = nn.Linear(1000,2)
</code></pre>
<p><strong>Training Code</strong></p>
<pre><code class="lang-auto">for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

print('Finished Training')
</code></pre>

ege_b: <p>I’m assuming you accounted for this issue in your <code>open_img</code> function. If you don’t mind sharing that one as well, I think I might be able to help you there. Although if that is not the case, then the issue seems to be that your input tensor has the size <code>[BxHxWxC]</code> whereas <code>[BxCxHxW]</code> aka <code>[32, 3, 256, 256]</code> was expected. You can simply <code>return image.permute(0, 3, 2, 1), labels</code> and it should work fine. Otherwise, run your code on debug mode and check the tensor sizes before-after every forward call.</p>

Naina_K: <p>I’ll try returning image.permute(0,3,2,1), labels<br>
Here’s my open_img function:</p>
<pre><code class="lang-auto">def open_img(f_path, n_channel=3, resize_to=None):
    if n_channel==1:
        # Load image in (256,256,1)
        im = np.expand_dims(cv2.imread(str(f_path), cv2.IMREAD_GRAYSCALE), axis=-1)
    else:
        im = cv2.imread(str(f_path))
    if resize_to is not None:
        im = cv2.resize(im, (resize_to, resize_to))
    im = (im/255.0).astype(np.float32)
    return im
</code></pre>

ege_b: <p>Yeah I think the format returned by <code>cv2.imread</code> differs from what a torch image tensor should look like and you did not account for that.</p>

Naina_K: <p>How should I change my function to account for this?</p>

ege_b: <p>Just add permute to your return, either in <code>open_img</code> which returns a <code>np.array</code> of shape <code>[HxWxC]</code> and in this case you have to change your return to <code>return im.permute(0, 3, 2, 1)</code> or the <code>__getitem__</code> return to the above one.</p>

Naina_K: <ol>
<li>
<p>for im.permute(0,3,2,1), I get an error saying: AttributeError: ‘numpy.ndarray’ object has no attribute ‘permute’</p>
</li>
<li>
<p>for image.permute(0,3,2,1), I get an error saying: RuntimeError: number of dims don’t match in permute</p>
</li>
</ol>

ege_b: <p>I’m sorry, the numpy method is not called permute, it is called transpose:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/809731e849f9f78f8f2cd4f6a10456b5a3a97d3b.png" data-download-href="https://discuss.pytorch.org/uploads/default/809731e849f9f78f8f2cd4f6a10456b5a3a97d3b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809731e849f9f78f8f2cd4f6a10456b5a3a97d3b_2_690x460.png" alt="image" data-base62-sha1="ilz35PQIJAQ5rCuvTSvuVcAH6Px" width="690" height="460" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809731e849f9f78f8f2cd4f6a10456b5a3a97d3b_2_690x460.png, https://discuss.pytorch.org/uploads/default/original/3X/8/0/809731e849f9f78f8f2cd4f6a10456b5a3a97d3b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/0/809731e849f9f78f8f2cd4f6a10456b5a3a97d3b.png 2x" data-dominant-color="F7F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">785×524 50.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<a href="https://numpy.org/doc/stable/reference/generated/numpy.transpose.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://numpy.org/doc/stable/reference/generated/numpy.transpose.html</a><br>
For both of them, <code>__getitem__</code> and <code>cv2.imread</code> do not directly return the batch, they return/read a single data point without a batch dimension. So try <code>return image.permute(2, 1, 0), labels</code> or <code>return im.transpose(axes=(2, 1, 0))</code> and it should work fine.</p>

Naina_K: <p>Thank you, I have fixed that error. However, when I go to train my model, I get the error: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cpu! (when checking argument for argument weight in method wrapper_nll_loss_forward)</p>
<pre><code class="lang-auto">for param in resnet50.parameters():
    param.requires_grad = False
    
resnet50.fc = nn.Linear(2048,2)
</code></pre>
<pre><code class="lang-auto">model = resnet50.to(device)
</code></pre>
<pre><code class="lang-auto"># Define the loss function and optimizer
criterion = torch.nn.CrossEntropyLoss(weight = class_weights)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
</code></pre>
<pre><code class="lang-auto">for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

print('Finished Training')
</code></pre>
<p>How do I fix this error?</p>

Manuel_Alejandro_Dia: <p>Make sure that your CrossEntropy weights are a CUDA tensor. This way when the loss function applies the weights they will be in the same device.</p>

Naina_K: <p>I have fixed the error above with the following code:</p>
<pre><code class="lang-auto"># Define the loss function and optimizer
criterion = torch.nn.CrossEntropyLoss(weight = class_weights.to(device))
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
</code></pre>
<p>However, when I go to train my model, I am consistently getting this output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6b1908c68a451a592cf731362c27e375bd67224.png" alt="training loss" data-base62-sha1="nMDGOC6cLePVoyO1O0VRB5YzzV2" width="300" height="389"></p>
<p><strong>training code</strong></p>
<pre><code class="lang-auto">for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, torch.max(labels, 1)[1])
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        print('Epoch: {} \tTraining Loss: {:.6f} '.format(epoch, running_loss))

print('Finished Training')
</code></pre>
<p>Where is this error coming from?</p>

Manuel_Alejandro_Dia: <p>First of all, maybe check the weight tensor. It is very weird that for CE loss you get zero.</p>
<p>Also, what are you trying to do with the following line? Why are you applying a max operation?:</p>
<pre><code class="lang-auto">loss = criterion(outputs, torch.max(labels, 1)[1])
</code></pre>
<p>In my opinion, your dataloader should give you the proper ground truth labels right away.</p>
<p>Try just running one debug pass with <code>batch_size</code> of 0 and see what loss value do you get.<br>
If you are trying to do semantic segmentation, also try to save the tensors and visualize them to make sure you are giving it the proper data to your <code>criterion</code>.</p>
<p>If you are trying to do multiclass segmentation, your tensors should be of <code>dtype=torch.long</code> and your output from the network should be of size <code>[B,C,H,W]</code> where B is batch size, C is number of classes to predict, and (H,W) is the original image size. The labels (ground truth) should come out of your dataloader with shape <code>[B, H, W]</code>.</p>

Naina_K: <p>Since my dataset is imbalanced, I used a weighted loss function and calculated the weights like this:</p>
<pre><code class="lang-auto">class_weights = [(class_cancer / d_train['cancer'].count()), (class_no_cancer / d_train['cancer'].count())]
class_weights=torch.tensor(class_weights,dtype=torch.float)

class_weights
</code></pre>
<p>if I simply use:</p>
<pre><code class="lang-auto">loss = criterion(outputs, labels)
</code></pre>
<p>I get an error saying: 0D or 1D target tensor expected, multi-target not supported</p>
<p>As for my task: my model needs to output 2 class scores (cancer, no cancer)</p>

Manuel_Alejandro_Dia: <p>Since your model needs to differentiate between two classes, the loss function best fitted for this should be <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>, because what you are describing is a binary classification task (it is either one class or the other).</p>
<p>Unless I have more than two classes in my dataset, I always define which one is the positive class (I would guess that it is ‘class_cancer’ in this case), and give it the weight you calculated with <code>class_cancer / d_train['cancer'].count()</code> as the positive weight.</p>

Naina_K: <p>I see that makes sense, but since the target and input size has to be the same for BCEWithLogitsLoss - how do I adjust my labels size to make it [32,2] (same as input size)?</p>