Weight tensor should be defined either for all or no classes

Shounak_Kundu: <p>Getting error as :</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-42-cafff645f983&gt; in &lt;module&gt;()
     13         print("targets[:, 0] size ==&gt; ",len(targets[:, 0]))
     14 
---&gt; 15         loss = criterion(outputs, targets[:, 0])
     16         loss.backward()
     17         optimizer.step()

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--&gt; 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

&lt;ipython-input-1-992294f0aa44&gt; in forward(self, outputs, targets)
     11 
     12     def forward(self, outputs, targets):
---&gt; 13         return self.loss(F.log_softmax(outputs), targets)

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--&gt; 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    145         _assert_no_grad(target)
    146         return F.nll_loss(input, target, self.weight, self.size_average,
--&gt; 147                           self.ignore_index, self.reduce)
    148 
    149 

/opt/anaconda/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1049         return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1050     elif dim == 4:
-&gt; 1051         return torch._C._nn.nll_loss2d(input, target, weight, size_average, ignore_index, reduce)
   1052     else:
   1053         raise ValueError('Expected 2 or 4 dimensions (got {})'.format(dim))

RuntimeError: weight tensor should be defined either for all or no classes at /opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THNN/generic/SpatialClassNLLCriterion.c:60
</code></pre>
<p><strong>Here is my code :</strong></p>
<p><code>weight = torch.ones(22)</code></p>
<p><code>criterion = CrossEntropyLoss2d(weight)</code></p>
<pre><code class="lang-auto">for epoch in range(1, num_epochs+1):
    epoch_loss = []
    iteration=1
    for step, (images, labels) in enumerate(trainLoader):
        print("Iter:"+str(iteration))
        iteration=iteration+1
        inputs = Variable(images)
        targets = Variable(labels)
        
        outputs = model(inputs)
        optimizer.zero_grad()
        print("outputs size ==&gt; ",len(outputs))
        print("targets[:, 0] size ==&gt; ",len(targets[:, 0]))

        loss = criterion(outputs, targets[:, 0])
        loss.backward()
        optimizer.step()
        epoch_loss.append(loss.data[0])
        
        average = sum(epoch_loss) / len(epoch_loss)
        
        print("loss: "+str(average)+" epoch: "+str(epoch)+", step: "+str(step))
</code></pre>
<p>Can you please help me here ?</p>
<p>Thanks in advance</p>

richard: <p>What are the sizes of the inputs to criterion? (ie, what is <code>outputs.size(), targets[:,0].size()</code>)?</p>
<p>The error implies that the size of the weights (22) isn’t equal to the number of classes in <code>outputs</code></p>

Shounak_Kundu: <p>Hi Richard , thanks a lot for your help.</p>
<p>I solved that one… I was giving class count wrongly.</p>
<p>My actual class count is 2.</p>
<p>So , I changed the weight size to 2.</p>
<p>But , now I am getting different error :</p>
<pre><code class="lang-auto">
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-62-a24d68a5b61a&gt; in &lt;module&gt;()
     25 #                 f'target (epoch: {epoch}, step: {step})')
     26 
---&gt; 27         loss = criterion(outputs, targets[:, 0])
     28         loss.backward()
     29         optimizer.step()

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--&gt; 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

&lt;ipython-input-1-62b1bfa509f1&gt; in forward(self, outputs, targets)
     11 
     12     def forward(self, outputs, targets):
---&gt; 13         return self.loss(F.log_softmax(outputs), targets)

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--&gt; 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    145         _assert_no_grad(target)
    146         return F.nll_loss(input, target, self.weight, self.size_average,
--&gt; 147                           self.ignore_index, self.reduce)
    148 
    149 

/opt/anaconda/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1049         return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1050     elif dim == 4:
-&gt; 1051         return torch._C._nn.nll_loss2d(input, target, weight, size_average, ignore_index, reduce)
   1052     else:
   1053         raise ValueError('Expected 2 or 4 dimensions (got {})'.format(dim))

RuntimeError: Assertion `cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes' failed.  at /opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THNN/generic/SpatialClassNLLCriterion.c:111
</code></pre>
<p>Can you please help ?</p>
<p>Thanks in advance</p>

ptrblck: <p>Now it seems, that your <code>target</code> has some illegal values.<br>
It’s either negative or has values &lt; <code>n_classes</code>.<br>
Could you check this?<br>
If you have two classes, your target should take values <code>[0, 1]</code>.</p>

windson: <p>Hi, am having the same problem… and my target has 3 classes, 0 and 1 and 255 which is the ignore index. I defined my weights as [0.1,0.9] for 0 and 1, but it didnt work… I suppose it s because of the ignore index 255, but isnt it gonna be ignored anyhow?<br>
How I can work around this? Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ptrblck: <p>This code snippet seems to work:</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss(
    ignore_index=255, weight=torch.tensor([1., 2.]), reduction='none')

x = torch.randn(3, 2)
y = torch.tensor([0, 1, 255])

loss = criterion(x, y)
print(loss)
&gt; tensor([0.8519, 1.1084, 0.0000])
</code></pre>

windson: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="16381">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CrossEntropyLoss</p>
</blockquote>
</aside>
<p>Thanks. But I have one further question.<br>
I have target with shape [n, w, h]  which n is the batch size;  output with shape [n, c, w, h]; and my mask is binary mask with ignore index 255.</p>
<p>I tried with your script and find it only works when the output channel dimension is 2, namely output shape [n,2,w,h] , one channel for 0, and one channel for 1.</p>
<p>It kinda makes sense coz the weight dimension should match the class dimension in the output, but just since it is a binary class problem, I wanna save some memory to output just 1 channel with 0 and 1 in the same channel. output [n,1,w,h]</p>
<p>Then to use the weights, do  I stil lhave to reshape it into 2 channel? or am I doing something wrong here.<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>For a binary classification you could either use your setup with two output channels and <code>nn.CrossEntropyLoss</code>, or alternatively you could output a single channel and use <code>nn.BCEWithLogitsLoss</code>.<br>
For former is used for a multi-class classification, while the latter approach is used for a binary or multi-label classification.</p>

windson: <p>Hi, Thanks for reply…</p>
<p>Yes, am aware of that.  am not using BCE loss, am using my own defined focalloss,which I used NLLLOSS inside.</p>
<pre><code class="lang-auto">class FocalLossnd(nn.Module):
    def __init__(self, weights=None, gamma=0, reduction='mean', ignore_idx=255):
        super().__init__()
        self.weights = weights
        self.gamma = gamma
        self.reduction= reduction
        self.eps = 1e-6
        self.ignore_idx = ignore_idx

    def forward(self, pred, target):
        target = target.type(torch.cuda.LongTensor)
        pt = torch.softmax(pred, dim=1)
        focal_weights = torch.pow((torch.ones(1).to(pred.device) - pt), self.gamma)
        focal = focal_weights * torch.log(pt + self.eps)

        criterion = nn.NLLLoss(
            weight=self.weights,
            ignore_index=self.ignore_idx,
            reduction=self.reduction)
        loss = criterion(focal, target)

        return loss
</code></pre>

Ruchi_Joshi: <h2><a name="i-am-getting-the-below-error-if-anyone-can-help-with-this-thanks-1" class="anchor" href="#i-am-getting-the-below-error-if-anyone-can-help-with-this-thanks-1"></a>I am getting the below error if anyone can help with this. Thanks!</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
14     <span class="hashtag-raw">#YOUR</span> CODE HERE<br>
15     predNN = nn(X_train_T)#??   # Forward pass<br>
—&gt; 16     error = loss(predNN, y_train_l.squeeze())#??    # find the loss<br>
17     optimizer.zero_grad()              # clear the gradients<br>
18     error.backward()              # Send loss backward</p>
<p>~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1193                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1194             return forward_call(*input, **kwargs)<br>
1195         # Do not call functions when jit is used<br>
1196         full_backward_hooks, non_full_backward_hooks = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>~\anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)<br>
1172<br>
1173     def forward(self, input: Tensor, target: Tensor) → Tensor:<br>
 → 1174         return F.cross_entropy(input, target, weight=self.weight,<br>
1175                                ignore_index=self.ignore_index, reduction=self.reduction,<br>
1176                                label_smoothing=self.label_smoothing)</p>
<p>~\anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)<br>
3024     if size_average is not None or reduce is not None:<br>
3025         reduction = _Reduction.legacy_get_string(size_average, reduce)<br>
 → 3026     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)<br>
3027<br>
3028</p>
<p>RuntimeError: weight tensor should be defined either for all 20 classes or no classes but got weight tensor of shape: [8]</p>
<p>My code is as mentioned below:</p>
<pre><code class="lang-auto">class NeuralNet(N.Module):
    def __init__(self, input_size, Nh1, Nh2, output_size):
        super(NeuralNet, self).__init__()
        self.fc1 = N.Linear(input_size, Nh1)
        self.fc2 = N.Linear(Nh1, Nh2)
        self.fc3 = N.Linear(Nh2, output_size)
        self.relu = N.ReLU()

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.fc3(x)
        return x

class LinearNet(N.Module):
    def __init__(self, input_size, output_size):
        super(LinearNet, self).__init__()
        self.fc = N.Linear(input_size, output_size)

    def forward(self, x):
        x = self.fc(x)
        return x

# Scaling features to have a maximum value of 1; grayscale images have value between 0 and 255
featureScale = 255
Nfeatures = Xtrain.shape[1]
X_train_T = Xtrain/featureScale
X_train_T = torch.tensor(X_train_T.astype(np.float32)).to(device)

# Converting to one hot vectors
y_train_l = torch.tensor(ytrain).to(device)
y_train_T = torch.nn.functional.one_hot(y_train_l)

Nclasses = np.max(ytrain)+1

nn = NeuralNet(Nfeatures,Nclasses,20,20).to(device)
sm = N.Softmax(dim=1)

#Weight the cross entropy loss to balance the classes
Nsamples_per_class = y_train_T.sum(axis=0)
Weight = Nsamples_per_class.sum()/Nsamples_per_class
loss = torch.nn.CrossEntropyLoss(weight=Weight)
learning_rate = 0.01
#YOUR CODE HERE
optimizer= optim.SGD(nn.parameters(), lr=learning_rate) # define optimizer

for epoch in range(20000):
    
    #YOUR CODE HERE
    predNN = nn(X_train_T)#??   # Forward pass
    error = loss(predNN, y_train_l.squeeze())#??    # find the loss
    optimizer.zero_grad()              # clear the gradients
    error.backward()              # Send loss backward
    optimizer.step()              # update weights 

    if(np.mod(epoch,5000)==0):
        print("Error =",error.detach().cpu().item())
        fig,ax = plt.subplots(1,2,figsize=(12,4))
        ax[0].plot(y_train_T[0:40].detach().cpu())
        ax[1].plot(sm(predNN[0:40]).detach().cpu())
        plt.show()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc480e6d0fa39ab7382d312371faf8286a1ee675.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc480e6d0fa39ab7382d312371faf8286a1ee675" title="Er1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc480e6d0fa39ab7382d312371faf8286a1ee675_2_690x406.png" alt="Er1" data-base62-sha1="qRC8XkfsXhPiUGhOvY5yIf4vCa9" width="690" height="406" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc480e6d0fa39ab7382d312371faf8286a1ee675_2_690x406.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc480e6d0fa39ab7382d312371faf8286a1ee675_2_1035x609.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc480e6d0fa39ab7382d312371faf8286a1ee675_2_1380x812.png 2x" data-dominant-color="F4F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Er1</span><span class="informations">1406×829 76.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on the posted code snippet you are creating a model returning logits for <code>20</code> classes while the error message indicates that the <code>weight</code> argument to <code>nn.CrossEntropyLoss</code> only contains 8 values.</p>
<p>It also seems you might be initializing the model wrong, since this code:</p>
<pre><code class="lang-python">Nclasses = np.max(ytrain)+1
nn = NeuralNet(Nfeatures,Nclasses,20,20).to(device)
</code></pre>
<p>passes <code>Nclasses</code> as the second argument, which is then used as <code>Nh1</code> in:</p>
<pre><code class="lang-python">class NeuralNet(N.Module):
    def __init__(self, input_size, Nh1, Nh2, output_size):
</code></pre>
<p>instead of <code>output_size</code>.</p>
<p>Could you double check this line of code, please?</p>