AttributeError: 'tuple' object has no attribute 'log_softmax'

Elena_Safrygina: <p>Hi,<br>
Could you please help me to find why following error happens.<br>
When I try to train model I get massage AttributeError: ‘tuple’ object has no attribute ‘log_softmax’<br>
at this line:<br>
loss = criterion(output, target)</p>
<p>And indeed, when I print output, I have tuple like this:<br>
(tensor([[ 0.3656, -0.2767],<br>
…<br>
[ 0.3415, -0.4962]], grad_fn=)</p>
<p>My input is [16,100] when 16 is number of features, 100 is batch size</p>
<p>How can I get rid of tuple?<br>
Below is how my model is defined.<br>
Sorry for my night English.<br>
Thank you very much in advance!</p>
<pre><code>N_INPUTS = 16
N_NEURONS = 5
N_OUTPUTS = 2
n_layers=2
drop_prob=0.85

class ImageRNN(nn.Module):
    def __init__(self, batch_size, n_inputs, n_neurons,drop_prob, n_outputs,n_layers):
        super(ImageRNN, self).__init__() 
        self.n_neurons = n_neurons#hidden_size
        self.batch_size = batch_size
        self.n_inputs = n_inputs
        self.n_outputs = n_outputs
        self.n_layers=n_layers
        self.drop_prob = drop_prob
        self.lstm = nn.LSTM(self.n_inputs, self.n_neurons, self.n_layers, 
                        dropout=drop_prob, batch_first=False)
        self.dropout = nn.Dropout(drop_prob)
    
        self.FC = nn.Linear(self.n_neurons, self.n_outputs)
    

def init_hidden(self):
    weight = next(self.parameters()).data
    
    hidden = (weight.new(self.n_layers, self.batch_size, self.n_neurons).zero_(),
                  weight.new(self.n_layers, self.batch_size, self.n_neurons).zero_())
    
    return hidden


    
def forward(self, X):
    X = X.unsqueeze(dim=0)
    self.hidden = self.init_hidden()     
    r_output, hidden = self.lstm(X, self.hidden)
    out = self.dropout(r_output)
    out = self.FC(out)
    out=out.view(-1, self.n_outputs)
    return out, self.hidden
</code></pre>
<p>model = ImageRNN(batch_size, N_INPUTS, N_NEURONS, drop_prob, N_OUTPUTS,n_layers)<br>
print(model)</p>

ptrblck: <p>I guess you are passing your output directly to your criterion, which seems to be <code>nn.CrossEntropyLoss</code>.<br>
Note that your model returns two outputs, i.e. <code>out</code> and <code>self.hidden</code>.<br>
You could assign two output variables to your model’s output:</p>
<pre><code class="lang-python">output, hidden = model(data)
</code></pre>
<p>or pass <code>output[0]</code> to your criterion.</p>

Dwayne_Wang: <p>Hi,<br>
I have a similar problem as above and I assigned output and hidden separately, but I still get the same error of the log_softmax. I’m using CrossEntropy as well.<br>
class RNN(nn.Module):</p>
<pre><code>def __init__(self, num_classes, input_size, hidden_size, num_layers,output_size):
    super(RNN, self).__init__()

    self.num_classes = num_classes
    self.num_layers = num_layers
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.out = torch.nn.Linear(hidden_size, input_size)

    self.rnn = nn.RNN(4, 5,2, batch_first=True)
    self.fc = nn.Linear(hidden_size, output_size)

def forward(self, x):
    # Initialize hidden and cell states
    # (num_layers * num_directions, batch, hidden_size) for batch_first=True
    batch_size = x.size(0)
    hidden = self.init_hidden(batch_size)
    out, hidden = self.rnn(x, hidden)
    
    out = out.contiguous().view(-1, self.hidden_size)
    out = self.fc(out)
    return out, hidden

def init_hidden(self, batch_size):
    hidden = torch.zeros(self.num_layers, batch_size, self.hidden_size)
    return hidden</code></pre>

ptrblck: <p>Could you post the code, which raises the error, please?<br>
<code>out</code> and <code>hidden</code> should both be tensors, so I’m not sure where the <code>tuple</code> is coming from.</p>

Dwayne_Wang: <p>So the error occurred at the torch.max line during the training. Would you please help me on it? Thank you very much.</p>
<pre><code class="lang-auto">X = torch.Tensor(train_input_list)
class RNN(nn.Module):
def __init__(self, num_classes, input_size, hidden_size, num_layers,output_size):
    super(RNN, self).__init__()

    self.num_classes = num_classes
    self.num_layers = num_layers
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.out = torch.nn.Linear(hidden_size, input_size)

    self.rnn = nn.RNN(4, 5,2, batch_first=True)
    self.fc = nn.Linear(hidden_size, output_size)

def forward(self, x):

    batch_size = x.size(0)
    hidden = self.init_hidden(batch_size)
    out, hidden = self.rnn(x, hidden)
    
    out = out.view(-1, self.hidden_size)
    out = self.fc(out)
    
    return out, hidden
def init_hidden(self, batch_size):
    hidden = torch.zeros(self.num_layers, batch_size, self.hidden_size)
    return hidden

rnn = RNN(num_classes, input_size, hidden_size, num_layers,1)
loss_func = nn.CrossEntropyLoss()
for epoch in range(num_epoch):
     Y_pred = rnn(X)
     out, predicted = torch.max(F.softmax(Y_pred,1), 1)```</code></pre>

ptrblck: <p><code>Y_pred</code> will be a <code>tuple</code>, since your model returns two tensors, <code>out</code> and <code>hidden</code>.<br>
Assuming you would only like to use <code>out</code> to calculate the prediction, you could use:</p>
<pre><code class="lang-python">out, predicted = torch.max(F.softmax(Y_pred[0], 1), 1)
</code></pre>
<p>Unrelated to this error, but note, that <code>nn.CrossEntropyLoss</code> expects raw logits as the model output, so you should not apply <code>softmax</code> or <code>max</code> on the output to calculate the loss. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
I assume you are using this line of code for debugging/printing only.</p>

Dwayne_Wang: <p>I calculate my loss like this<br>
loss = loss_func(Y_pred, Y)<br>
Yeah, the error did occur when calculating the loss. So would you please suggest how should I solve the problem?</p>

ptrblck: <p><code>Y_pred</code> will be a <code>tuple</code>, already mentioned. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
If you want to use the <code>out</code> tensor as the model output, you should use <code>loss_func(Y_pred[0], Y)</code>.</p>

Dwayne_Wang: <p>Thank you very much. This error has been fixed now. But when I try running my code, it seems that there is some problem with CrossEntropy similar to what you solved previously: IndexError: Target 1 is out of bounds… I think defined the data types right, but it still does not work somehow.</p>

ptrblck: <p>How did you specify <code>output_shape</code> while creating the model?<br>
The <code>output_shape</code> should be the number of classes you are working with and the target should contain indices in the range <code>[0, nb_classes-1]</code>.</p>

Dwayne_Wang: <p>Yes, that is the problem. Thank you so much.</p>

Jyoti_Rawat: <p>I am also getting the same problem while prediction.  Could you please help:</p>
<p>test_dl = DataLoader(tst_data, batch_size=64, shuffle=False)<br>
test = []<br>
print(‘Predicting on test dataset’)<br>
for batch, _ in tst_data:<br>
<span class="hashtag">#test_dl:</span><br>
<span class="hashtag">#batch</span> = batch.permute(0, 2, 1)<br>
batch=batch.to(device)<br>
print(batch.shape)<br>
out = model.to(device)<br>
y_hat = F.log_softmax(out, dim=1).argmax(dim=1) ### at this line I am receiving an error.</p>
<pre><code>test += y_hat.tolist()
</code></pre>
<p>class LSTMClassifier(nn.Module):</p>
<pre><code>def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
    super().__init__()
    self.hidden_dim = hidden_dim
    self.layer_dim = layer_dim
    self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
    self.fc = nn.Linear(hidden_dim, output_dim)
    self.batch_size = None
    self.hidden = None
  

def forward(self, x):
    h0, c0 = self.init_hidden(x)
   
    print(x.size())
    out, (hn, cn) = self.lstm(x, (h0, c0))
    out = self.fc(out[:, -1, :])
    return out

def init_hidden(self, x):
    h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
    c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)

    print(h0.shape)
    print(x.size(0))
    print(layer_dim)
    return [t.to(device) for t in (h0, c0)]
</code></pre>
<p>Thank you in advance!</p>
<p>Error :: AttributeError: ‘LSTMClassifier’ object has no attribute ‘log_softmax’</p>

ptrblck: <p>Based on the error message it seems that you are trying to call <code>log_softmax</code> on the <code>LSTMClassifier</code> directly:</p>
<pre><code class="lang-python">model = LSTMClassifier(args)
model.log_softmax
</code></pre>
<p>which won’t work.<br>
In your code snippet you are however using <code>F.log_softmax</code>, so could you check, if you’ve replaced <code>import torch.nn.functional as F</code> with <code>F = model</code>?</p>

Jyoti_Rawat: <p>I was not calling model.train and directly calculating the y_hat.So, I changed to -</p>
<p>test_dl = DataLoader(tst_data, batch_size=64, shuffle=False)<br>
test = []<br>
print(‘Predicting on test dataset’)<br>
model = model.to(device)<br>
for batch, _ in tst_data:<br>
batch=batch.to(device)<br>
out = model.train()(batch)<br>
y_hat = F.log_softmax(out, dim=1).argmax(dim=1)</p>
<h2>
<a name="however-i-got-another-issue-not-torchutilsdatadatasettensordataset-object-at-0x7f0d61a87490-predicting-on-test-dataset-torchsize1-4000-500-4000-1-torchsize4000-1" class="anchor" href="#however-i-got-another-issue-not-torchutilsdatadatasettensordataset-object-at-0x7f0d61a87490-predicting-on-test-dataset-torchsize1-4000-500-4000-1-torchsize4000-1"></a>However, I got another issue not:<br>
&lt;torch.utils.data.dataset.TensorDataset object at 0x7f0d61a87490&gt;<br>
Predicting on test dataset<br>
torch.Size([1, 4000, 500])<br>
4000<br>
1<br>
torch.Size([4000])</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
6 for batch, _ in tst_data:<br>
7     batch=batch.to(device)<br>
----&gt; 8     out = model.train()(batch)<br>
9     y_hat = F.log_softmax(out, dim=1).argmax(dim=1)<br>
10</p>
<p>5 frames<br>
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in check_input(self, input, batch_sizes)<br>
201             raise RuntimeError(<br>
202                 ‘input must have {} dimensions, got {}’.format(<br>
 → 203                     expected_input_dim, input.dim()))<br>
204         if self.input_size != input.size(-1):<br>
205             raise RuntimeError(</p>
<p>RuntimeError: input must have 3 dimensions, got 1</p>
<p>Could you please look.</p>

ptrblck: <p>The new error is raised by an RNN, as a 3-dimensional input is required as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html?highlight=rnn#torch.nn.RNN">docs</a> so you would need to pass it as <code>[seq_len, batch_size, features]</code> in the default setting or <code>[batch_size, seq_len, features]</code>, if <code>batch_first</code> is set to <code>True</code>.</p>

Jyoti_Rawat: <p>Thank you. It works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Wlanin: <pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/pxg/DAN/DANet/experiments/segmentation/train.py", line 282, in &lt;module&gt;
    trainer.training(epoch)
  File "/home/pxg/DAN/DANet/experiments/segmentation/train.py", line 214, in training
    loss = self.criterion(outputs, target)
  File "/home/pxg/DAN/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pxg/DAN/DANet/encoding/parallel.py", line 130, in forward
    return self.module(inputs, *targets[0], **kwargs[0])
  File "/home/pxg/DAN/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pxg/DAN/DANet/encoding/nn/loss.py", line 68, in forward
    return super(SegmentationLosses, self).forward(*outputs)
  File "/home/pxg/DAN/venv/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/pxg/DAN/venv/lib/python3.6/site-packages/torch/nn/functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/pxg/DAN/venv/lib/python3.6/site-packages/torch/nn/functional.py", line 1295, in log_softmax
    ret = input.log_softmax(dim)
AttributeError: 'tuple' object has no attribute 'log_softmax'
</code></pre>
<p>Hello, can you help me with my question? Thank you very much.</p>

ptrblck: <p><code>input</code> seems to be a <code>tuple</code> while you are expecting a tensor on which you could use <code>.log_softmax</code>. I’m not familiar with your use case and thus don’t know what <code>input</code> contains but you might want to unwrap or index it instead before using <code>log_softmax</code>.</p>

Wlanin: <pre><code class="lang-auto"> def forward(self, *inputs):
        if not self.se_loss and not self.aux:
            return super(SegmentationLosses, self).forward(*inputs)
        elif not self.se_loss:
            pred1, pred2, target = tuple(inputs)
            loss1 = super(SegmentationLosses, self).forward(pred1, target)
            loss2 = super(SegmentationLosses, self).forward(pred2, target)
            return loss1 + self.aux_weight * loss2
</code></pre>
<p>Thank you for your reply. This is the last wrong position. Can you help me see where the problem is?</p>

ptrblck: <p>The previous error points to:</p>
<pre><code class="lang-python">loss = self.criterion(outputs, target)
</code></pre>
<p>which might be used inside one of the <code>forward</code> methods so you would have to check them.</p>