RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'mat2'

aklagoo: <p>I’m trying to implement a skip-gram model. When I pass the input vector to the hidden layer, I get the error:<br>
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument <span class="hashtag">#2</span> ‘mat2’</p>
<pre><code class="lang-auto">ONE_HOT_SIZE  = len(dictionary)
EMBEDDING_DIM = 200
EPOCH_NUM     = 5

def one_hot(index):
    one_hot_array = torch.zeros(ONE_HOT_SIZE, dtype=torch.long)
    one_hot_array[index] = 1.
    return one_hot_array

class SkipGramModel(nn.Module):
    def __init__(self, one_hot_size, embedding_dim):
        super(SkipGramModel, self).__init__()
        self.hidden = nn.Linear(one_hot_size, embedding_dim)
        self.linear_out = nn.Linear(embedding_dim, one_hot_size)
    def forward(self, x):
        embed = self.hidden(x)
        out   = self.linear_out(embed)
        out   = F.log_softmax(out, dim=0)
        return out

model         = SkipGramModel(ONE_HOT_SIZE, EMBEDDING_DIM)
loss_function = nn.NLLLoss()
optimizer     = optim.SGD(model.parameters(), lr=0.01)

losses = []
total_count = EPOCH_NUM * len(targets)

with tqdm(total = total_count) as pbar:
    for epoch in range(EPOCH_NUM):
        total_loss = 0
        for target in targets:
            # Convert input word and target to one-hot vectors
            word   = one_hot(target[0])
            target = one_hot(target[1])

            # Reset gradient
            model.zero_grad()

            # Get the output from the model
            prediction = model(word)
            
            # Calculate loss
            loss = loss_function(prediction, target)

            # Optimize
            loss.backward()
            optimizer.step()

            # Add loss to total loss
            total_loss += loss

            # Update progress bar
            pbar.update(1)
        losses.append(total_loss)
</code></pre>
<p><strong>Error message:</strong></p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-39-884ed7fa9d4b&gt; in &lt;module&gt;
     11 
     12             # Get the output from the model
---&gt; 13             prediction = model(word)
     14 
     15             # Calculate loss

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

&lt;ipython-input-36-238e5eb99042&gt; in forward(self, x)
      5         self.linear_out = nn.Linear(embedding_dim, one_hot_size)
      6     def forward(self, x):
----&gt; 7         embed = self.hidden(x)
      8         out   = self.linear_out(embed)
      9         out   = F.log_softmax(out, dim=0)

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     90     @weak_script_method
     91     def forward(self, input):
---&gt; 92         return F.linear(input, self.weight, self.bias)
     93 
     94     def extra_repr(self):

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1406         ret = torch.addmm(bias, input, weight.t())
   1407     else:
-&gt; 1408         output = input.matmul(weight.t())
   1409         if bias is not None:
   1410             output += bias

RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'mat2'
</code></pre>

ptrblck: <p><code>word</code> is probably a <code>LongTensor</code>, which will yield this error.<br>
You can transform it to a <code>FloatTensor</code> by calling <code>.float()</code> on it:</p>
<pre><code class="lang-python">word = word.float()
prediction = model(word)
</code></pre>
<p>Not relevant for this error, but which loss function are you using?</p>

aklagoo: <p>I’m using the NLLLoss function. After making the change you suggested, I get the error “Expected two dimensions, got one”</p>

ptrblck: <p>Could you print the shape or <code>words</code>?<br>
<code>nn.Linear</code> expects the input as a tensor of shape <code>[batch_size, *, in_feautres]</code>.</p>
<p>Also, <code>nn.NLLLoss</code> expects a <code>LongTensor</code> as the target containing the class indices. It should therefore not be one-hot encoded.</p>

aklagoo: <p>I messed up by feeding the training set one by one. I’ll feed batches instead. Thanks for the help!<br>
And why shouldn’t the vector consist of one-hots?<br>
Shouldn’t each class be true or false instead?</p>

ptrblck: <p>No, as given in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss">docs</a>, the loss expects class indices instead:</p>
<blockquote>
<p>The target that this loss expects should be a class index in the range [0,C−1] where C = number of classes; if ignore_index is specified, this loss also accepts this class index (this index may not necessarily be in the class range).</p>
</blockquote>
<p>You can see the applied formula in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">CrossEntropyLoss docs</a>. Note that <code>nn.CrossEntropyLoss</code> will internally use <code>log_softmax</code> + <code>NLLLoss</code>.</p>

aklagoo: <p>I read it. I’ll pass the target without encoding it. Thanks a bunch!</p>

tzchao: <p>I thought the error stated that it was expecting a <code>LongTensor</code> instead of a <code>FloatTensor</code>. I put .float() to the corresponding place and the error disappeared.<br>
Why is this error message indicating it is actually expecting a <code>FlaotTensor</code>?<br>
Thanks!</p>

ptrblck: <p>The <code>RuntimeError</code> claims that the first argument to the operation is the “right” one and the second argument has the wrong data type.<br>
In this case the second argument is the <code>weight</code> as seen in the stack trace, which explains the error message.</p>