Two tower networks stops learning when adding linear layers

username: Mimi_Lazarova
cooked: <p>I have been trying to train a relatively simple two-tower net for recommendation. I am using PyTorch and the implementation is the following: basically embeddings layers for users and items, optional feed-forward net for both towers, dot product between the user and items representations, and sigmoid.<br>
I am trining with Binary cross entropy loss and Adam optimizer. When I am using only the embeddings, I see improvements from epoch to epoch (loss is decreasing and the evaluation metric are increasing). However, once I add even a single feed-forward layer, the network learns just a bit in the first epoch and then stagnates. I have tried to had code one linear layer with ReLU, to check if the issue is with the way I am creating the list of layers, but this did not change anything.</p>
<p>Has anybody else had a similar problem?</p>
<p>Implementation:</p>
<pre><code class="lang-auto">class SimpleTwoTower(nn.Module):
    
    def __init__(self, n_items, n_users, ln):
        super(SimpleTwoTower, self).__init__()
        
        self.ln = ln
        self.item_emb = nn.Embedding(num_embeddings=n_items, embedding_dim=self.ln[0])
        self.user_emb = nn.Embedding(num_embeddings=n_users, embedding_dim=self.ln[0])
       
        
        self.item_layers = [] #nn.ModuleList()
        self.user_layers = [] #nn.ModuleList()
        
        for i, n in enumerate(ln[0:-1]):
            m = int(ln[i+1])
            self.item_layers.append(nn.Linear(n, m, bias=True))
            self.item_layers.append(nn.ReLU())
            
            self.user_layers.append(nn.Linear(n, m, bias=True))
            self.user_layers.append(nn.ReLU())
            
            
        self.item_layers = nn.Sequential(*self.item_layers)
        self.user_layers = nn.Sequential(*self.user_layers)
        
        self.dot = torch.matmul
        self.sigmoid = nn.Sigmoid()
        
    def forward(self, items, users):
        
        item_emb = self.item_emb(items)
        user_emb = self.user_emb(users)
        
        item_emb = self.item_layers(item_emb)
        user_emb = self.user_layers(user_emb)

        dp = self.dot(user_emb, item_emb.t())
        return self.sigmoid(dp)
</code></pre>

username: ptrblck
cooked: <p>I assume you are using <code>nn.BCELoss</code> based on your description and the usage of the <code>sigmoid</code> activation? If so, could you remove the sigmoid and replace the criterion with <code>nn.BCEWithLogitsLoss</code>, as it would give you more numerical stability?</p>
<p>Based on your usage of <code>self.dot = torch.matmul</code> it seems you are creating an output tensor of the shape <code>[batch_size, batch_size]</code>. I assume this is expected?</p>

username: Mimi_Lazarova
cooked: <p>I was indeed using <code> nn.BCELoss</code> with sigmoid activation. I tried with your suggestion, i.e. to remove the sigmoid and train with <code>nn.BCEWithLogitsLoss</code> but I did not get any improvement. This is the loss throughout training for 15 epochs.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f4cd230e77d5b46c12116f8804ccc9c089630e9.png" data-download-href="https://discuss.pytorch.org/uploads/default/1f4cd230e77d5b46c12116f8804ccc9c089630e9" title="Screenshot 2021-04-08 at 10.24.39"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f4cd230e77d5b46c12116f8804ccc9c089630e9_2_345x224.png" alt="Screenshot 2021-04-08 at 10.24.39" data-base62-sha1="4sToaWn4gPjyX2KOBQLlXCz1w37" width="345" height="224" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f4cd230e77d5b46c12116f8804ccc9c089630e9_2_345x224.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f4cd230e77d5b46c12116f8804ccc9c089630e9_2_517x336.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f4cd230e77d5b46c12116f8804ccc9c089630e9_2_690x448.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-08 at 10.24.39</span><span class="informations">776×504 9.58 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you have any idea why this may not be working?</p>
<p>To your second question - yes, the idea is to get a score for each user-item pair, so this is why the output tensors is of size <code>[batch_size, batch_size]</code>.</p>

username: ptrblck
cooked: <p>No idea yet, what might be wrong. However, it looks as if the loss is pretty stable.<br>
Could you check the gradients of all parameters after calling the <code>.backward()</code> operation via:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    print(name, param.grad.abs().sum())
</code></pre>
<p>and check their values?</p>

username: Mimi_Lazarova
cooked: <p>Ok, so the results are:</p>
<pre><code class="lang-auto">999 item_emb.weight tensor(0.0596, device='cuda:0')
999 user_emb.weight tensor(0.0473, device='cuda:0')
999 item_layers.0.weight tensor(0.3596, device='cuda:0')
999 item_layers.0.bias tensor(0.0378, device='cuda:0')
999 user_layers.0.weight tensor(0.4759, device='cuda:0')
999 user_layers.0.bias tensor(0.0414, device='cuda:0')
1999 item_emb.weight tensor(0.0013, device='cuda:0')
1999 user_emb.weight tensor(0.0053, device='cuda:0')
1999 item_layers.0.weight tensor(0.0060, device='cuda:0')
1999 item_layers.0.bias tensor(0.0005, device='cuda:0')
1999 user_layers.0.weight tensor(0.1563, device='cuda:0')
1999 user_layers.0.bias tensor(0.0128, device='cuda:0')
2999 item_emb.weight tensor(0., device='cuda:0')
2999 user_emb.weight tensor(0., device='cuda:0')
2999 item_layers.0.weight tensor(0., device='cuda:0')
2999 item_layers.0.bias tensor(0., device='cuda:0')
2999 user_layers.0.weight tensor(0., device='cuda:0')
2999 user_layers.0.bias tensor(0., device='cuda:0')
3999 item_emb.weight tensor(0., device='cuda:0')
3999 user_emb.weight tensor(0., device='cuda:0')
3999 item_layers.0.weight tensor(0., device='cuda:0')
3999 item_layers.0.bias tensor(0., device='cuda:0')
3999 user_layers.0.weight tensor(0., device='cuda:0')
3999 user_layers.0.bias tensor(0., device='cuda:0')
</code></pre>
<p>The number in front of each row is the update step. I guess this was what you thought was happening.</p>

username: Mimi_Lazarova
cooked: <p>I tried two appriaches: changing <code>nn.ReLU</code> to <code>nn.LeakyReLU</code> and separately, I added batch normalization layer after every activation. Both worked, and I think I will stick with the leaky ReLU approach.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Mimi_Lazarova" data-post="6" data-topic="117330">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mimi_lazarova/48/36810_2.png" class="avatar"> Mimi_Lazarova:</div>
<blockquote>
<p>Both worked, and I think I will stick with the leaky ReLU approach.</p>
</blockquote>
</aside>
<p>I assume this means your model trains now? If so, great to hear!</p>

username: Chenxi_He
cooked: <p>I have a similar issue. But it still doesn’t work with LeakyReLU or batch normalization.Furthermore the user tower predict same result for different users.</p>

username: ecdrid
cooked: <p>I don’t think the problem is with ReLU layers as such,</p>
<p>If we will execute the below,</p>
<pre><code class="lang-python">
import torch
import torch.nn as nn

class SimpleTwoTower(nn.Module):
    
    def __init__(self, n_items, n_users, ln):
        super(SimpleTwoTower, self).__init__()
        
        self.ln = ln
        self.item_emb = nn.Embedding(num_embeddings=n_items, embedding_dim=self.ln[0])
        self.user_emb = nn.Embedding(num_embeddings=n_users, embedding_dim=self.ln[0])
       
        
        self.item_layers = [] #nn.ModuleList()
        self.user_layers = [] #nn.ModuleList()
        
        for i, n in enumerate(ln[0:-1]):
            m = int(ln[i+1])
            self.item_layers.append(nn.Linear(n, m, bias=True))
            self.item_layers.append(nn.ReLU())
            
            self.user_layers.append(nn.Linear(n, m, bias=True))
            self.user_layers.append(nn.ReLU())
            
            
        self.item_layers = nn.Sequential(*self.item_layers)
        self.user_layers = nn.Sequential(*self.user_layers)
        
        self.dot = torch.matmul
        self.sigmoid = nn.Sigmoid()

    def forward(self, items, users):
        
        item_emb = self.item_emb(items)
        user_emb = self.user_emb(users)
        
        item_emb = self.item_layers(item_emb)
        user_emb = self.user_layers(user_emb)

        dp = self.dot(user_emb, item_emb.t())
        return self.sigmoid(dp)

</code></pre>
<pre><code class="lang-auto"># let's create our model here using the snippet as-is
SimpleTwoTower(10000, 1000,  [2**10, 2**8, 2**6])

"""
# output will be something like this

SimpleTwoTower(
  (item_emb): Embedding(10000, 1024)
  (user_emb): Embedding(1000, 1024)
  (item_layers): Sequential(
    (0): Linear(in_features=1024, out_features=256, bias=True)
    (1): ReLU()
    (2): Linear(in_features=256, out_features=64, bias=True)
    (3): ReLU() **#### &lt;&lt;&lt;&lt;&lt;&lt;&lt; isn't this an issue?**
  )
  (user_layers): Sequential(
    (0): Linear(in_features=1024, out_features=256, bias=True)
    (1): ReLU()
    (2): Linear(in_features=256, out_features=64, bias=True)
    (3): ReLU() **#### &lt;&lt;&lt;&lt;&lt;&lt;&lt; isn't this an issue?**
  )
  (sigmoid): Sigmoid()
)
"""
</code></pre>
<p>So, Do we really want to have that <code>ReLU</code> as well? From what i know about the Two-tower model, we will use the raw_output from the last linear layer as the feature vector (i.e. latent vec etc)</p>
<p>Also, Do we want to use embedding layers as well?</p>
<p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/mimi_lazarova">@Mimi_Lazarova</a></p>
<p>Please correct my understanding!</p>
<p>Be Safe,<br>
Best,<br>
Aditya.</p>

username: silpara
cooked: <p>I have a similar issue the two towers after training give very random results. How did you fix this issue?</p>