TypeError: forward() missing 1 required positional argument: 'target' when using AdaptiveLogSoftmaxWithLoss

username: Akib_Sadmanee
cooked: <p>I am trying to build a next word prediction model with pytorch in google colab. As my vocabulary size is over 1.5 million, I am using AdaptiveLogSoftmaxWithLoss module of pytorch to reduce RAM consumption.</p>
<p>The simple BiLSTM model definition is as follows:</p>
<pre><code class="lang-python">class BLSTM(nn.Module):
    def __init__(self, emb_size, hidden_size, num_layers, vocab_size, cutoffs):
        super(BLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.blstm = nn.LSTM(emb_size, hidden_size, num_layers, batch_first=True, bidirectional=True)
        self.fc = nn.AdaptiveLogSoftmaxWithLoss(hidden_size*2, vocab_size, cutoffs)
        # self.fc = nn.Linear(hidden_size*2, vocab_size)
    
    def forward(self, x):
        # Set initial states
        h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)

        # Forward propagate LSTM
        embed = nn.Embedding(vocab_size, emb_size)
        out, _ = self.blstm(embed(x), (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size*2)
        
        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :])
        
        return out
</code></pre>
<p>The model and loss function are called as follows:</p>
<pre><code class="lang-python">model = BLSTM(emb_size, hidden_size, num_layers, vocab_size, cutoffs)

# Loss and optimizer
criterion = nn.NLLLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
</code></pre>
<p>Inside the epoch the loss is calculated as follows:</p>
<pre><code class="lang-python">        inputs = x[0].to(device)
        targets = x[1].to(device)

        # Forward pass
        outputs = model(inputs)
        outputs = outputs.to(device)
        loss = criterion(outputs, targets)
        print(loss.item())
        
        #Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>This is the complete error trace I am getting after running the epoch loop:</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-33-51e15380f8c7&gt; in &lt;module&gt;()
      8 
      9         # Forward pass
---&gt; 10         outputs = model(inputs)
     11         outputs = outputs.to(device)
     12         loss = criterion(outputs, targets)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--&gt; 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() missing 1 required positional argument: 'target'
</code></pre>
<p><strong>I tried the same code with a simple nn.Linear() layer and the code runs fine. But when I replace the Linear layer with AdaptiveLogSoftmaxWithLoss, I get the above mentioned error.</strong></p>

username: Unity05
cooked: <p>Hi <a class="mention" href="/u/akib_sadmanee">@Akib_Sadmanee</a></p>
<p>according to <code>AdaptiveLogSoftmaxWithLoss</code>'s name and to its implementation (<a href="https://pytorch.org/docs/master/_modules/torch/nn/modules/adaptive.html#AdaptiveLogSoftmaxWithLoss" rel="nofollow noopener">https://pytorch.org/docs/master/_modules/torch/nn/modules/adaptive.html#AdaptiveLogSoftmaxWithLoss</a>), it also requires the target argument for calculating the loss. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Regards,<br>
Unity05</p>

username: Akib_Sadmanee
cooked: <p>Hello <a class="mention" href="/u/unity05">@Unity05</a><br>
Thank you for your reply. Can you help me on how to pass the target. Below is the class description,<br>
<strong>torch.nn.AdaptiveLogSoftmaxWithLoss` ( <em>in_features: int, n_classes: int, cutoffs: Sequence[int], div_value: float = 4.0, head_bias: bool = False</em> )</strong><br>
I don’t see any parameter that takes in the targets tensor.</p>

username: Unity05
cooked: <p>Hi <a class="mention" href="/u/akib_sadmanee">@Akib_Sadmanee</a>,</p>
<p>Indeed, the constructor doesn’t take <code>target</code> as an argument.</p>
<pre><code class="lang-auto">def __init__(
        self,
        in_features: int,
        n_classes: int,
        cutoffs: Sequence[int],
        div_value: float = 4.,
        head_bias: bool = False
    ) -&gt; None:
</code></pre>
<p>Therefore, your initialization is right.</p>
<p>However, when using <code>AdaptiveLogSoftmaxWithLoss</code>'s <code>forward()</code> method (what you do in your own <code>forward()</code> method) you can see by checking its implementation that it expects <code>target</code> as an argument.</p>
<pre><code class="lang-auto"> def forward(self, input: Tensor, target: Tensor) -&gt; _ASMoutput:
</code></pre>
<p>I hope this helped. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Regards,<br>
Unity05</p>

username: vdw
cooked: <p><code>AdaptiveLogSoftmaxWithLoss</code> returns a loss, so you do not need another <code>NLLoss</code>. I never used that one, but I assume you need to change your <code>forward()</code> method to</p>
<pre><code class="lang-auto">def forward(self, x, targets):
        # Set initial states
        h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)

        # Forward propagate LSTM
        embed = nn.Embedding(vocab_size, emb_size)
        out, _ = self.blstm(embed(x), (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size*2)
        
        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :], targets)       
        return out
</code></pre>
<p>Note that <code>out</code> is</p>
<pre><code class="lang-auto">NamedTuple`` with ``output`` and ``loss`` fields:
            * **output** is a Tensor of size ``N`` containing computed target
              log probabilities for each example
            * **loss** is a Scalar representing the computed negative
              log likelihood loss
</code></pre>
<p>And you need to change your training loop to</p>
<pre><code class="lang-auto"># Forward pass
outputs = model(inputs, targets)
outputs = outputs.to(device) # the loss is "somehwere" in the outputs tuple
#loss = criterion(outputs, targets)
#print(loss.item())
</code></pre>

username: Akib_Sadmanee
cooked: <p>Thanks <a class="mention" href="/u/vdw">@vdw</a>. It’s working now. For clarification,<br>
I didn’t find any optimizer in the AdaptiveLogSoftmaxWithLoss. I still need to run the optimizer, right?<br>
And for backpropagation, I assume now I need to call outputs.loss.backward().</p>

username: Akib_Sadmanee
cooked: <p>Thanks. This solution works. I tried the sample mentioned above by vdw which does the same thing you mentioned.</p>