Indexing embeddings, matrix factorization

jbuddy_13: <p>Hello all,</p>
<p>I’m trying to execute a modified version of matrix factorization via deep learning. The goal is to find a matrix of shape [9724x300] where the rows are items and there are (arbitrarily) 300 features. The function would be optimized when the dot product of vector_i and vector_j are really, really close to the value in the interaction matrix, Xij.</p>
<p>Xij has dimensions [9724x9724] where the value in cell [0,1] is the number of users who liked both items i and j. Ergo, when optimized, Vi*VjT should be really close to the number of users who liked both items i and j.</p>
<p>I’ve modified this code from a <a href="https://blog.fastforwardlabs.com/2018/04/10/pytorch-for-recommenders-101.html" rel="nofollow noopener">tutorial</a>. The key difference is that in this resource, the author has a user-to-item matrix, not an item-to-item matrix.</p>
<p>I’m stuck trying to index vectors i and j in tensor self.vectors. It appears the datatype did not match what was expected, despite making i and j into LongTensors.</p>
<p>Any feedback would be appreciated!</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

class MatrixFactorization(torch.nn.Module):
    def __init__(self, n_items=len(movie_ids), n_factors=300):
        super().__init__()
        
        self.vectors = nn.Embedding(n_items, n_factors,sparse=True)
        

    def forward(self, i,j):
        return (self.vectors([i])*torch.transpose(self.vectors([j]))).sum(1)

    def predict(self, i, j):
        return self.forward(i, j)

model = MatrixFactorization(n_items= len(movie_ids),n_factors=300)
loss_fn = nn.MSELoss() 
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for i in range(len(movie_ids)):
    for j in range(len(movie_ids)):
    # get user, item and rating data
        rating = Variable(torch.FloatTensor([Xij[i, j]]))
        # predict
        i = Variable(torch.LongTensor([int(i)]))
        j = Variable(torch.LongTensor([int(j)]))
        prediction = model(i, j)
        loss = loss_fn(prediction, rating)

        # backpropagate
        loss.backward()

        # update weights
        optimizer.step()
</code></pre>
<p>And the error I receive:</p>
<pre><code class="lang-auto">TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not list
</code></pre>

albanD: <p>Hi,</p>
<p>Variables are not needed anymore. and predict methods should not be needed. An updated version of your code is:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

class MatrixFactorization(torch.nn.Module):
    def __init__(self, n_items=len(movie_ids), n_factors=300):
        super().__init__()
        
        self.vectors = nn.Embedding(n_items, n_factors,sparse=True)
        

    def forward(self, i,j):
        # i and j are LongTensors here of size (batch)
        feat_i = self.vectors(i)
        feat_j = self.vectors(j)
        # feat_i and feat_j are of size (batch, n_factors)
        # Since you only want the interactions element-wise
        # for i and j in the batch dimension, you want diag(feat_i * feat_j.t())
        # This can be efficiently computed using element-wise product and summing
        result = (feat_i * feat_j).sum(-1)
        # result is of size (batch)
        return result


model = MatrixFactorization(n_items= len(movie_ids),n_factors=300)
loss_fn = nn.MSELoss() 
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for i in range(len(movie_ids)):
    for j in range(len(movie_ids)):
    # get user, item and rating data
        rating =torch.FloatTensor([Xij[i, j]])
        # predict
        i = torch.LongTensor([int(i)])
        j = torch.LongTensor([int(j)])
        prediction = model(i, j)
        loss = loss_fn(prediction, rating)

        # Reset the gradients to 0
        optimizer.zero_grad()

        # backpropagate
        loss.backward()

        # update weights
        optimizer.step()
</code></pre>
<p>Note that with this code you can do real batch size and have:</p>
<pre><code class="lang-auto"># Generate a batch of 4 pairs i,j
i = torch.LongTensor([1, 3, 4, 5])
j = torch.LongTensor([2, 4, 6, 7])

# Get the ground truth and do forward for all of them at once
ratings = X[i, j]
pred = model(i, j)
# By default, MSELoss with compute the average MSELoss in the batch (you can change that if your need, check the doc how to do so)
loss = loss_fn(pred, ratings)
</code></pre>

jbuddy_13: <p>Thank you much for your detailed response!<br>
Can you elaborate on the last block of code beginning at</p>
<pre><code class="lang-auto"># Generate a batch of 4 pairs i,j
</code></pre>
<p>Does this code block replace the nested for loop?<br>
And what is the significance of the pairs, just an example? Or will this sample large batches of data?</p>
<pre><code class="lang-auto">i = torch.LongTensor([1, 3, 4, 5])
j = torch.LongTensor([2, 4, 6, 7])
</code></pre>
<p>Thanks <a class="mention" href="/u/alband">@albanD</a>!</p>

albanD: <p>Pytorch has been built to work with batch of data. That allows to perform larger operations and thus better use device such as GPUs.<br>
A batch is basically a bunch of independent data that you process at the same time.</p>
<p>In your case, if <code>len(movie_ids) == 2</code>, you have 4 pairs of indices to evaluate: <code>(0,0), (0,1), (1,0), (1,1)</code>.<br>
You can do this using your nested for loops or by doing a single forward with a batch of 4 samples with <code>i = torch.LongTensor([0, 0, 1, 1])</code> and <code>j=torch.LongTensor([0, 1, 0, 1])</code>. That way, in one call to your model, you get the result for all four samples, instead of doing 4 calls to your model with the for loops.<br>
Loss functions such as MSE are built to support such examples and you can see in <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MSELoss" rel="nofollow noopener">the doc</a> that it has a special argument <code>reduction</code> to choose how the individual values for each element in your batch should be aggregated into the final loss value.</p>

jbuddy_13: <p>Awesome! I tried the batch method you’ve described, and the following warning was returned:</p>
<pre><code class="lang-auto">/anaconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py:431: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([5931640])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
</code></pre>
<p>However, the code hasn’t thrown an error (yet) so maybe all is well.</p>

albanD: <p>Check the size of the elements you give to your loss.<br>
One has the full batch size, but the other is 1. They should both be of size batch size. So something is wrong somewhere <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

jbuddy_13: <p>Found it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Thanks again for your help!</p>