Torch.argmax returns a tensor containing all zeros?

farazk86: <p>Ive trained my segmentation network using a U-Net model. I want to now visualize the result from the network output.</p>
<p>Theshape of output is <code>[1,2,256,256]</code>. One channel for background and the other for foreground. Here is what I get for <code>print(output.detach().squeeze())</code>:</p>
<pre><code class="lang-auto">tensor([[[ 6.1109,  6.1109,  6.1126,  ...,  6.1580,  6.1317,  6.1317],
         [ 6.1109,  6.1109,  6.1126,  ...,  6.1580,  6.1317,  6.1317],
         [ 6.1125,  6.1125,  6.1139,  ...,  6.1281,  6.1075,  6.1075],
         ...,
         [ 8.3482,  8.3482,  8.3518,  ...,  8.7196,  8.7172,  8.7172],
         [ 8.3446,  8.3446,  8.3485,  ...,  8.7216,  8.7194,  8.7194],
         [ 8.3446,  8.3446,  8.3485,  ...,  8.7216,  8.7194,  8.7194]],

        [[-6.8949, -6.8949, -6.8821,  ..., -7.1064, -7.0877, -7.0877],
         [-6.8949, -6.8949, -6.8821,  ..., -7.1064, -7.0877, -7.0877],
         [-6.9068, -6.9068, -6.8939,  ..., -7.0928, -7.0780, -7.0780],
         ...,
         [-8.3679, -8.3679, -8.3719,  ..., -8.7457, -8.7435, -8.7435],
         [-8.3656, -8.3656, -8.3700,  ..., -8.7487, -8.7467, -8.7467],
         [-8.3656, -8.3656, -8.3700,  ..., -8.7487, -8.7467, -8.7467]]])
</code></pre>
<p>To get the predictions from this output I did:</p>
<p><code>predicted = torch.argmax(output.detach(), 1).squeeze(0)</code></p>
<p>But this gives a tensor with all zeros??</p>
<p>I then write this image along with the respective label using:</p>
<pre><code class="lang-auto">f_ = str(idx) + '_predicted.jpg'
io.imwrite(os.path.join(output_folder, f_), predicted.float().cpu().numpy())

f_ = str(idx) + '_target.jpg'
io.imwrite(os.path.join(output_folder, f_), labels.squeeze().float().cpu().numpy())
</code></pre>
<p>The shape of <code>predicted</code> and <code>label</code>(after squeeze())  is <code>[256 256]</code>.</p>
<p>Label is being written fine as can be seen:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/25acec536dd71d51e6fe932c9117656b64d56536.jpeg" alt="0_target" data-base62-sha1="5ni9snaGhzOoiYAW5pj0OBrvLRc" width="256" height="256"></p>
<p>But I am only getting black for predicted:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/292c28c9005e0c26c3c520f3cae696397832e93f.jpeg" alt="0_predicted" data-base62-sha1="5SebBykYHbdCt3v0fzw2Ullzimb" width="256" height="256"></p>
<p>Am I doing this right? How else am I supposed to visualize the network output?</p>
<p>Thanks</p>

farazk86: <p>Strange thing is <code>np.amax(predicted.numpy())</code> gives me <code>0</code>. So how is there no maximum value recorded in <code>predicted</code>?</p>

ptrblck: <p>I think the maximal values might be in channel0. In your example <code>output</code> tensor, it looks like channel0 has positive values, while channel1 seems to only have negative values.<br>
At least in the numbers you’ve posted.</p>
<p>Could this be the case, i.e. your model just predicts background for all pixels?</p>

farazk86: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Even in that case I should get some output from <code>argmax</code> instead of a zero tensor. The way I am visualizing the outputs here was suggested by you in my previous question: <a href="https://discuss.pytorch.org/t/understanding-how-to-label-target-tensors-for-3d-volumes/30101" class="inline-onebox">Understanding how to label/target tensors for 3D volumes</a></p>
<p>And this seems to work fine for 3D volumes. But here I am using 2D images. But the concept should be the same and it should work. As I have labeled my label tensor during training such that channel0 represents background and channel1 represents my foreground.</p>
<p>So shouldnt I be looking for the maximum values in <code>dim=1</code> ?</p>

ptrblck: <p>Yes your approach is right and I still think that the zero tensor is a valid answer, if your model just overfits to class0.</p>
<p>Have a look at this example:</p>
<pre><code class="lang-python">x = torch.cat((torch.ones(1, 1, 5, 5), torch.zeros(1, 1, 5, 5)), 1)
print(x)
&gt; tensor([[[[1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.]],

           [[0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.]]]])
print(torch.argmax(x, 1))
&gt; tensor([[[0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0]]])
</code></pre>
<p>A zero tensor is in this case the right answer, since all values in channel0 are larger than in channel1.<br>
If we manipulate a certain value in channel1, you’ll also get the right answer:</p>
<pre><code class="lang-python">x[0, 1, 0, 0] = 2
print(torch.argmax(x, 1))
&gt; tensor([[[1, 0, 0, 0, 0],
           [0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0]]])
</code></pre>

Elahe_Heydari: <p>Hi,<br>
I want to get argmax of my preds in batch.when I print argmax in model, It return values, but when I save the perds in list, it return zeros for all elements in list. I don’t know why it is happend, how can I get values in list?</p>
<p>my code is:</p>
<pre><code class="lang-auto">for epoch in tqdm(range(1, num_epochs + 1)):
    # start_time = time.time()
    num_examples =0
    scheduler.step()
    correct_pred = 0
    lr = scheduler.get_lr()[0]

    model.eval()
    train_loss_total = 0.0
    num_steps = 0
    test_loss_total = 0
    num = 0

    ### Training
    for i, batch in enumerate(test1_loader):

        input_samples, gt_samples = batch[2], batch[3]
        var_input =input_samples
        var_gt = gt_samples
        optimizer.zero_grad()

        preds = model(var_input)
        var_gt = var_gt.reshape(8,64,64,64)
        var_gt = var_gt.long()

        # print(preds.shape)
        loss1 = criterion(preds, var_gt)   
        # loss.backward(retain_graph=True)
        loss1.backward()
        optimizer.step()
        test1_loss_total += loss1.item()
        num_steps += 1

        _, predicted_test = torch.max(preds.data, 1)
        total_test1 += var_gt.nelement()
        correct_test1 += predicted_test.eq(var_gt.data).sum().item()
        print('************************************************************************')

        print(preds.shape)
        _, v = torch.max(preds, dim=0)
        print(v.unique())
        _, v1 = torch.max(var_gt, dim=0)
        print('   v1     ',v1.unique())


        print('----------------')

        if epoch == num_epochs:
           predslist.append(preds)
           var_gtlist.append(var_gt)
           print('done')

    test1_accuracy = 100 * correct_test1 / total_test1
    print('Epoch {}, test Loss: {:.3f}'.format(epoch, test1_loss_total / num_steps),
          "Testing Accuracy: %d %%" % (test1_accuracy))

    print("finished training")
</code></pre>
<p>and result is:</p>
<pre><code class="lang-auto">Epoch 4, test Loss: 0.148 Testing Accuracy: 99 %
finished training
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 3, 4, 5, 6, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 3, 4, 5, 6, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([2, 3, 4, 5, 6, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 3, 4, 5, 6, 7])
   v1      tensor([0, 3, 7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 4, 5, 6, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 3, 4, 5, 6, 7])
   v1      tensor([2, 5, 7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 6, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 3, 4, 5, 6, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 3, 4, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
tensor([0, 1, 2, 3, 4, 5, 6, 7])
   v1      tensor([7])
----------------
done
************************************************************************
torch.Size([8, 21, 64, 64, 64])
100%|██████████| 5/5 [21:29&lt;00:00, 257.86s/it]tensor([0, 1, 2, 3, 4, 5, 6, 7])
   v1      tensor([7])
----------------
done
Epoch 5, test Loss: 0.185 Testing Accuracy: 99 %
finished training
</code></pre>
<p>my code for list is:</p>
<pre><code class="lang-auto">for ii in predslist:
  for j in range(0, batch_size):
      om = torch.argmax(ii[j].squeeze(), dim=0)
      print (om.unique()) 
</code></pre>
<p>that it return zeros.</p>

ptrblck: <p>To get the class predictions <code>torch.argmax</code> or <code>torch.max</code> should be called with the <code>dim</code> argument specifying the “class dimension”, i.e. the dimension which is set to the number of classes and contains the logits for each class.</p>
<p>In your training code you are using <code>dim=0</code>. Are you sure the class dimension is in dim0, not the batch dimension?</p>
<p>The last code snippet iterates the <code>predslist</code> and calls <code>argmax</code> again on <code>dim=0</code>, which will return zeros for a single sample in the batch.</p>
<p>For a vanilla classification use case your output would have the shape <code>[batch_size, nb_classes]</code> and you should therefore call <code>torch.argmax(output, dim=1)</code> as shown in my example.</p>
<p>PS: the usage of <code>.data</code> is not recommended, as it might have unwanted side effects.</p>

Elahe_Heydari: <p>thanks for your reply.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="34161">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>In your training code you are using <code>dim=0</code> . Are you sure the class dimension is in dim0, not the batch dimension?</p>
</blockquote>
</aside>
<p>yes, it is for batch dimension, I made mistake.<br>
Size of each element of predslist is torch.Size([21, 64, 64, 64]).I get argmax of dim=0 that it returns just zeros.<br>
i wrote this code that shows values of each element of predslist:</p>
<pre><code class="lang-auto">for ii in predslist:
  for j in range(0, batch_size):
      print(ii[j])
      print('ii[j].shape:',ii[j].shape)
      om = torch.argmax(ii[j].squeeze(), dim=0).detach().cpu().numpy()
      print ('np.unique(om):' ,np.unique(om)) 
      print('om.shape:',om.shape)
      print('************************************************************')
</code></pre>
<p>and some of outputs is:</p>
<pre><code class="lang-auto">ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [0]
om.shape: (64, 64, 64)
************************************************************
tensor([[[[ 42.3527,  42.4556,  42.4556,  ...,  30.8631,  29.5867,  28.1223],
          [ 42.4102,  42.9640,  42.9640,  ...,  30.8631,  29.5867,  28.1223],
          [ 42.4102,  42.9640,  42.9640,  ...,  30.8631,  29.5867,  28.1223],
          ...,
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783],
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783],
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783]],

         [[ 44.3158,  44.3947,  44.3947,  ...,  31.4897,  31.1392,  30.3788],
          [ 44.5054,  44.7882,  44.7882,  ...,  31.6356,  31.1392,  30.3788],
          [ 44.5054,  44.7882,  44.7882,  ...,  31.6356,  31.1392,  30.3788],
          ...,
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783],
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783],
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783]],

         [[ 45.5911,  45.5911,  45.5911,  ...,  32.1989,  32.1989,  31.6583],
          [ 45.7082,  45.7082,  45.7082,  ...,  32.5894,  32.4413,  31.6583],
          [ 45.7082,  45.7082,  45.7082,  ...,  32.6178,  32.4413,  31.6583],
          ...,
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783],
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783],
          [ 48.3376,  48.3376,  48.3376,  ...,  48.6783,  48.6783,  48.6783]],

         ...,

         [[138.6365, 140.9799, 142.6059,  ...,  58.8695,  58.8695,  58.8695],
          [144.4590, 144.4590, 144.4590,  ...,  58.8695,  58.8695,  58.8695],
          [144.4590, 144.4590, 144.4590,  ...,  58.8695,  58.8695,  58.8695],
          ...,
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309],
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309],
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309]],

         [[ 82.9900,  83.1118,  83.7028,  ...,  50.3198,  50.3198,  50.3198],
          [ 86.4008,  86.4008,  86.4008,  ...,  50.3198,  50.3198,  50.3198],
          [ 86.4008,  86.4008,  86.4008,  ...,  50.3198,  50.3198,  50.3198],
          ...,
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309],
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309],
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309]],

         [[ 46.8168,  46.8168,  46.8168,  ...,  47.2092,  47.2092,  47.2092],
          [ 46.8168,  46.8168,  46.8168,  ...,  47.2092,  47.2092,  47.2092],
          [ 46.8168,  46.8168,  46.8168,  ...,  47.2092,  47.2092,  47.2092],
          ...,
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309],
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309],
          [ 47.3822,  47.3822,  47.3822,  ...,  48.2309,  48.2309,  48.2309]]],


        [[[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         ...,

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]]],


        [[[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         ...,

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]]],


        ...,


        [[[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         ...,

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]]],


        [[[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         ...,

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]]],


        [[[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         ...,

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]],

         [[  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          ...,
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000],
          [  0.0000,   0.0000,   0.0000,  ...,   0.0000,   0.0000,   0.0000]]]],
       grad_fn=&lt;SelectBackward&gt;)

ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [0]
om.shape: (64, 64, 64)
************************************************************
</code></pre>
<p>As you see predslist has values, but the argmax in dim0 is zeros.<br>
I want to decode preds and I must have nonzeros values.<br>
How can I get nonzeros values for argmax?<br>
The shape of output of argmax should be (64,64,64).Did I make a mistake somewhere in my code?<br>
Also, If I print argmax for dim=1,2,3, it has nonzeros values for elements of predslist !!!</p>

ptrblck: <p>I’m still unsure how you are calculating the predictions.<br>
Could you post the corrected code again and explain what each dimension means, please?</p>

Elahe_Heydari: <p>yes,sure.<br>
I trained my dataset by following network that must extract 21 features. but, I’m not sure that it works correct or not.<br>
I guess the zeros value of arg max is because of my model. when I return out2 in model, I get zeroes value, but when I change network like this, the argmax have nonzeros values!!!<br>
when I use this model, the accuracy will be very low!!!</p>
<pre><code class="lang-auto">for epoch in tqdm(range(1, num_epochs + 1)):
    # start_time = time.time()
    num_examples =0
    scheduler.step()
    correct_pred = 0
    lr = scheduler.get_lr()[0]

    model.eval()
    train_loss_total = 0.0
    num_steps = 0
    test_loss_total = 0
    num = 0

    ### Training
    for i, batch in enumerate(test1_loader):

        input_samples, gt_samples = batch[2], batch[3]
        var_input =input_samples
        var_gt1 = gt_samples
        optimizer.zero_grad()

        preds = model(var_input)
        var_gt = var_gt1.reshape(8,64,64,64)
        var_gt = var_gt.long()

        # print(preds.shape)
        loss1 = criterion(preds, var_gt)   
        # loss.backward(retain_graph=True)
        loss1.backward()
        optimizer.step()
        test1_loss_total += loss1.item()
        num_steps += 1

        _, predicted_test = torch.max(preds, 1)
        total_test1 += var_gt.nelement()
        correct_test1 += predicted_test.eq(var_gt).sum().item()


        if epoch == num_epochs:
            
     
            print('***************************************************')
            predslist.append(preds)
            var_gtlist.append(var_gt1)
            print('done')
 
    test1_accuracy = 100 * correct_test1 / total_test1
    print('Epoch {}, test Loss: {:.3f}'.format(epoch, test1_loss_total / num_steps),
          "Testing Accuracy: %d %%" % (test1_accuracy))
    print("finished training")
</code></pre>
<p>and:<br>
I  calculate argmax for dim0 that it is for 21 classes.</p>
<pre><code class="lang-auto">for ii in predslist:
  for j in range(0, batch_size):
      # print(ii[j])
      print('ii[j].shape:',ii[j].shape)
      om = torch.argmax(ii[j].squeeze(), dim=0).detach().cpu().numpy()
      print ('np.unique(om):' ,np.unique(om)) 
      print('om.shape:',om.shape)
      print('************************************************************')
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 0  1  5  7  9 10 12 13 14 15 16 18 20]
om.shape: (64, 64, 64)
************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 0  1  7  9 10 12 13 14 15 16 18 20]
om.shape: (64, 64, 64)
************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 0  1  2  5  7  9 10 11 13 14 16 18 19 20]
om.shape: (64, 64, 64)
************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 0  1  5  7 10 11 13 14 15 16 18 19 20]
om.shape: (64, 64, 64)
************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 0  1  5  7 10 11 12 13 14 15 16 18 19 20]
om.shape: (64, 64, 64)
************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 1  2  5  7 10 13 14 16 18 19 20]
om.shape: (64, 64, 64)
************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 0  1  2  4  5  7 10 11 12 13 14 16 18 19 20]
om.shape: (64, 64, 64)
************************************************************
ii[j].shape: torch.Size([21, 64, 64, 64])
np.unique(om): [ 1  2  5  7 10 11 13 14 16 18 20]
om.shape: (64, 64, 64)
</code></pre>
<p>and this is my model:</p>
<pre><code class="lang-auto">class Myconv(nn.Module):
    def __init__(self):
        super(Myconv, self).__init__()
        self.conv1 = nn.Conv3d(in_channels=1, out_channels=10, kernel_size=(3, 3, 3),stride=1, padding=1)
        self.norm1 =nn.BatchNorm3d(num_features = 10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.max1 = nn.MaxPool3d(kernel_size=(3, 3 ,3), stride=1, padding=1)


        self.conv2 = nn.Conv3d(in_channels=10, out_channels=21, kernel_size=(3, 3, 3),
                               stride=1, padding=1 )
        self.norm2 =nn.BatchNorm3d(num_features = 21, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.max2 = nn.MaxPool3d(kernel_size=(3,3,3), stride=1, padding=1)

        self.conv3 = nn.Conv3d(in_channels=21, out_channels=21, kernel_size=(3, 3, 3),stride=1, padding=1)
        self.norm3 =nn.BatchNorm3d(num_features = 21, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.max3 = nn.MaxPool3d(kernel_size=(3, 3 ,3), stride=1, padding=1)

    def forward(self, x):
        out1 = self.conv1(x)
        out1 = self.norm1(out1)
        out1 = F1.relu(out1)
        out1 = self.max1(out1)

        out2 = self.conv2(out1)
        out2 = self.norm2(out2)
        out2 = F1.relu(out2)
        out2 = self.max2(out2)

        out3 = self.conv3(out2)
        out3 = self.norm3(out3)
        out3 = F1.relu(out3)
        out3 = self.max3(out3)
        return out3
</code></pre>
<p>when I edit paremeters of model and get more accuracy (99%), I get zeroes argmax!!!</p>

Hamzeh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I have few tesnors with size <code> torch.Size([4, 11, 256, 256])</code>, whit unique values <code>[0, 1, 2, 3, 4, 5, 6, 7, 8 ,9, 10]</code>, when I use <code>torch.argmax(mask, dim=1)</code>, it just returns zero with <code>torch.Size([1])</code><br>
What is the problem?</p>
<p><code>mask = tensor([[[[5., 5., 5.,  ..., 4., 4., 4.],           [5., 5., 5.,  ..., 4., 4., 4.],           [5., 5., 5.,  ..., 4., 4., 4.],           ...,           [0., 0., 0.,  ..., 4., 4., 4.],           [0., 0., 0.,  ..., 4., 4., 4.],           [0., 0., 0.,  ..., 4., 4., 4.]], dtype=torch.float64)</code></p>
<pre><code class="lang-auto">mask = torch.unique(torch.argmax(mask, dim=1))
# print(mask.shape, mask)
# torch.Size([1]) tensor([0])
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Hamzeh" data-post="12" data-topic="34161">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamzeh/48/50569_2.png" class="avatar"> Hamzeh:</div>
<blockquote>
<p>when I use <code>torch.argmax(mask, dim=1)</code>, it just returns zero with <code>torch.Size([1])</code></p>
</blockquote>
</aside>
<p>This shouldn’t happen as only one dimension should be reduced:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randint(0, 11, [4, 11, 256, 256])
out = torch.argmax(x, dim=1)
print(out.shape)
# torch.Size([4, 256, 256])
</code></pre>
<p>In the following code you are additionally using <code>torch.unique</code> so did you check if the output of <code>torch.argmax</code> returns all zeros (which would be the case if this dim contains the max values)?</p>

Hamzeh: <aside class="quote no-group" data-username="ptrblck" data-post="13" data-topic="34161">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>you are additionally using <code>torch.unique</code> so did you check if the output of <code>torch.argmax</code> returns all zeros (which would be the case if this dim contains the max valu</p>
</blockquote>
</aside>
<p>Yes, output of <code>torch.argmax</code> is all zeros, I used <code>torch.unique</code> to make sure that I have all classes (i.e. values are between 0 and 10), but I noticed it just returns zeros. I used a random input and it works totally fine.<br>
<code>mask = tensor([[[[3., 3., 2.,  ..., 1., 1., 1.],           [3., 3., 2.,  ..., 1., 1., 1.],           [3., 2., 2.,  ..., 1., 1., 1.],           ...,           [2., 2., 2.,  ..., 3., 3., 3.],           [2., 2., 2.,  ..., 3., 3., 3.],           [2., 2., 2.,  ..., 2., 2.,   [[3., 3., 2.,  ..., 1., 1., 1.],           [3., 3., 2.,  ..., 1., 1., 1.],           [3., 2., 2.,  ..., 1., 1., 1.],           ...,           [2., 2., 2.,  ..., 3., 3., 3.],           [2., 2., 2.,  ..., 3., 3., 3.],           [2., 2., 2.,  ..., 2., 2.,            [[3., 3., 2.,  ..., 1., 1., 1.],           [3., 3., 2.,  ..., 1., 1., 1.],           [3., 2., 2.,  ..., 1., 1., 1.],           ...,           [2., 2., 2.,  ..., 3., 3., 3.],           [2., 2., 2.,  ..., 3., 3., 3.],           [2., 2., 2.,  ..., 2., 2., 2.]], </code><br>
<code># torch.Size([1, 11, 256, 256])</code></p>
<pre><code class="lang-auto">mask = torch.argmax(mask, dim=1)
# tensor([[[0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         ...,
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0]]])
# torch.Size([1, 256, 256])
</code></pre>

ptrblck: <p>Based on the results could you check if the <code>mask</code> contains the largest values in <code>channel0</code>?</p>
<pre><code class="lang-python">x = torch.randint(0, 11, [1, 11, 256, 256])
# make sure channel0 contains the largest values
x[:, 0] += 100000
out = torch.argmax(x, dim=1)

print((out == 0).all())
# tensor(True)
</code></pre>

Hamzeh: <aside class="quote no-group" data-username="ptrblck" data-post="15" data-topic="34161">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>print((out == 0).all())</code></p>
</blockquote>
</aside>
<p>I noticed all channels (i.e. 0-10) has been repeated</p>
<pre><code class="lang-auto">mask[0, 0] = 
tensor([[3., 3., 2.,  ..., 1., 1., 1.],
        [3., 3., 2.,  ..., 1., 1., 1.],
        [3., 2., 2.,  ..., 1., 1., 1.],
        ...,
        [2., 2., 2.,  ..., 3., 3., 3.],
        [2., 2., 2.,  ..., 3., 3., 3.],
        [2., 2., 2.,  ..., 2., 2., 2.]])

mask[0, 1] = tensor([[3., 3., 2.,  ..., 1., 1., 1.],
        [3., 3., 2.,  ..., 1., 1., 1.],
        [3., 2., 2.,  ..., 1., 1., 1.],
        ...,
        [2., 2., 2.,  ..., 3., 3., 3.],
        [2., 2., 2.,  ..., 3., 3., 3.],
        [2., 2., 2.,  ..., 2., 2., 2.]])
</code></pre>
<p>I understood the issue.<br>
Thank you for your help</p>