[Noob] Model with Embedding bag not learning, problem with my model?

username: Stacie_Davis
cooked: <p>Hi,</p>
<p>Apologies if my model itself is downright trash, please correct in that case. I am just learning pytorch.<br>
So, I am trying to do sentiment analysis on imdb dataset. 1-gram, sending the word index to the embedding-bag, then single fc layer.</p>
<pre><code class="lang-auto">
class ffn(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedBag = nn.EmbeddingBag(1000, 20,sparse=True)
        self.fc = nn.Linear(20, 1)
        
    def forward(self, x):
        x = self.embedBag(x)
        y = self.fc(x)
        return t.sigmoid(y)

 
mod = ffn()
#using batch_size 1 so that no padding is required
imdb_data = t.utils.data.DataLoader(imdb_ds(train_x, train_y),batch_size=1)
optim = t.optim.Adagrad(mod.parameters(), lr=0.001 )

loss_ls = []
for i,(x,y) in enumerate(imdb_data):
    optim.zero_grad()
    ybar = mod(x)
    loss = F.binary_cross_entropy(ybar,t.tensor(y,dtype=t.float32))
    loss_ls.append(loss.item())
    loss.backward()
    optim.step()
    if i%5000==0: print(loss)
</code></pre>
<p>Is there a problem with loss? Is sigmoid giving problem? My loss is just stuck around 0.7, randomly fluctuating.</p>

username: ptrblck
cooked: <p>It’s usually better to remove the sigmoid and use <code>nn.BCEWithLogitsLoss</code> or <code>F.binary_cross_entropy_with_logits</code>.<br>
However, your model might be too small, so I would recommend to add a relu and another linear layer to it.<br>
Let me know, if that helps. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Stacie_Davis
cooked: <p>Thanks for your suggestion. It really helped <img src="https://discuss.pytorch.org/images/emoji/apple/relaxed.png?v=9" title=":relaxed:" class="emoji" alt=":relaxed:"></p>
<p>2 questions, if you don’t mind:</p>
<ol>
<li>I used binary cross with logits. But then when I wrote my function to find accuracy, I had to take sigmoid of the output of the model. So, isn’t it better to directly put sigmoid or softmax in model itself? When does the reverse become more useful?</li>
<li>Why have functions like sigmoid, tanh have been moved from nn.functional to torch.sigmoid, torch.tanh? Them being inside nn.functional makes kinda more sense intuitively, that functions are inside functional. Pretty sure team pytorch has better reasons than mine <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:">
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>Yes, you could use a sigmoid to get the probabilities and calculate the accuracy using a threshold. Alternatively you could also use a threshold directly on the logits (no sigmoid), but a threshold in the range <code>[0, 1]</code> might be easier and more intuitive to use.<br>
If you put the sigmoid directly into the model, <code>nn.BCELoss</code> will apply <code>torch.log</code> internally, while passing logits to <code>nn.BCEWithLogitsLoss</code> will use the <a href="https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations">log-sum-exp trick</a> as seen in <a href="https://github.com/pytorch/pytorch/blob/35cdb785228b8abea1d3bdb844aa5980e6642f8d/aten/src/ATen/native/Loss.cpp#L202-L218">these lines of code</a> and will thus yield more numerical stability.</p>
</li>
<li>
<p>I don’t have a strong opinion on it and feel free to chime in your opinion on what’s the less confusing approach. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Generally “mathematical” functions should go into the <code>torch</code> namespace, while NN-specific methods should stay in the <code>nn.functional</code> namespace as described <a href="https://github.com/pytorch/pytorch/issues/22117">here</a>. <code>torch.nn.functional.sigmoid</code> should still work I think.</p>
</li>
</ol>

username: Stacie_Davis
cooked: <p>As the Sigmoid/Relu are still inside torch.nn, I feel it is a bit confusing.<br>
I feel they should be moved to torch.math, and these sigmoid/relu should be moved to torch.math.functional.</p>

username: ptrblck
cooked: <p>I wouldn’t create a new <code>math</code> namespace, as this would mean that also <code>matmul</code>, <code>dot</code>, <code>sum</code> etc. should be moved there, wouldn’t it?<br>
Feel free to add your suggestions to the linked GitHub issue to discuss it further. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>