I can't overfit on a simple batch with LSTM applied to Time Series data

username: Yasser
cooked: <p>Hello <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> ,</p>
<p>I am new to PyTorch, and I built an LSTM model with Embeddings to predict a target of size 720 using time series data with a sequence of length 14 and more than 18 000 features (which includes date related data).<br>
The problem is that my model always outputs the average of all the labels he saw during training, so to make it sure it’s true, I tried to overfit my model on a single batch of size 5 and the results confirm what I thought:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2e540370ed7ffbb7ddb3446c147ee10c15c2db8.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2e540370ed7ffbb7ddb3446c147ee10c15c2db8" title="Capture d’écran 2020-04-21 à 11.01.50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2e540370ed7ffbb7ddb3446c147ee10c15c2db8_2_690x30.png" alt="Capture d’écran 2020-04-21 à 11.01.50" data-base62-sha1="u5Fr3Jd9fBZ1VlWhz57gdlIMLS8" width="690" height="30" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2e540370ed7ffbb7ddb3446c147ee10c15c2db8_2_690x30.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2e540370ed7ffbb7ddb3446c147ee10c15c2db8_2_1035x45.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2e540370ed7ffbb7ddb3446c147ee10c15c2db8_2_1380x60.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2020-04-21 à 11.01.50</span><span class="informations">1636×72 29.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ee79be891032888314c5a5a6fe15a59137f079c.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ee79be891032888314c5a5a6fe15a59137f079c" title="Capture d’écran 2020-04-21 à 11.02.24"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ee79be891032888314c5a5a6fe15a59137f079c_2_690x25.png" alt="Capture d’écran 2020-04-21 à 11.02.24" data-base62-sha1="koc1i5A1PZeKUl2ly46AZDHm2Kg" width="690" height="25" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ee79be891032888314c5a5a6fe15a59137f079c_2_690x25.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ee79be891032888314c5a5a6fe15a59137f079c_2_1035x37.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ee79be891032888314c5a5a6fe15a59137f079c_2_1380x50.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2020-04-21 à 11.02.24</span><span class="informations">1700×64 29.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The two screenshots above show the predictions of the model regarding the 86th component of the five targets of the batch, and the same for the 57th one, as you can see it’s nearly the average that’s predicted.</p>
<p>This is the code of the model :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab01ba5b256133764a4c5fc25e3f52861b22f5cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/ab01ba5b256133764a4c5fc25e3f52861b22f5cc" title="Capture d’écran 2020-04-21 à 11.55.08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab01ba5b256133764a4c5fc25e3f52861b22f5cc_2_387x499.png" alt="Capture d’écran 2020-04-21 à 11.55.08" data-base62-sha1="ooNlFKe2nzxHkTeA3iIFAPfnBZq" width="387" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab01ba5b256133764a4c5fc25e3f52861b22f5cc_2_387x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab01ba5b256133764a4c5fc25e3f52861b22f5cc_2_580x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab01ba5b256133764a4c5fc25e3f52861b22f5cc_2_774x998.png 2x" data-dominant-color="313132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2020-04-21 à 11.55.08</span><span class="informations">1316×1698 311 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is how I made the training :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2001d4429cc5f08ca8f9a2f905650cf6c7a85c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/b2001d4429cc5f08ca8f9a2f905650cf6c7a85c6" title="Capture d’écran 2020-04-21 à 12.13.39"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2001d4429cc5f08ca8f9a2f905650cf6c7a85c6_2_394x500.png" alt="Capture d’écran 2020-04-21 à 12.13.39" data-base62-sha1="poFey4YjUSjjikn6S08eQIthbfM" width="394" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2001d4429cc5f08ca8f9a2f905650cf6c7a85c6_2_394x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2001d4429cc5f08ca8f9a2f905650cf6c7a85c6_2_591x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2001d4429cc5f08ca8f9a2f905650cf6c7a85c6_2_788x1000.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2020-04-21 à 12.13.39</span><span class="informations">1316×1668 316 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/9611b5c1f02261b73abe344c476db79f244afdef.png" data-download-href="https://discuss.pytorch.org/uploads/default/9611b5c1f02261b73abe344c476db79f244afdef" title="Capture d’écran 2020-04-21 à 12.14.09"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9611b5c1f02261b73abe344c476db79f244afdef_2_345x126.png" alt="Capture d’écran 2020-04-21 à 12.14.09" data-base62-sha1="lpzyYccrMRcuM2S6CSOogN8xRMz" width="345" height="126" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9611b5c1f02261b73abe344c476db79f244afdef_2_345x126.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9611b5c1f02261b73abe344c476db79f244afdef_2_517x189.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9611b5c1f02261b73abe344c476db79f244afdef_2_690x252.png 2x" data-dominant-color="313232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2020-04-21 à 12.14.09</span><span class="informations">862×316 22.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>1 - I don’t know what I am doing wrong since the model can’t predict exactly for each input, but always gives the output of the targets in the training set.<br>
The L1 Loss doesn’t decrease to 0 obviously but get stuck around 0,98 for the training data</p>
<p>2- I would also like to know If the Embeddings are very well used in this case, because I tried to encode the two variables related to datetime (week and day) as you can see in the code above, and concatenated them with the remaining features to feed them after to the LSTM.<br>
In the first Embedding for the weeks, I go from a vocabulary of size 52 to an embedded vector of size 28.<br>
In the second one for the days, I go from a vocabulary of size 7 to an embedded vector of size 4.</p>
<p>If there’s anything wrong in the code, please don’t hesitate to mention it <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> , maybe that’s the reason my model can’t overfit, and learn very well.</p>
<p>NB : For the input that I gave to the model, it’s a sequence of length 14 which represent 14 successive days where I give to the model all the features WITH their corresponding target in the input, and for the corresponding output it’s the target of the day AFTER,</p>
<p>Thank you very much ! <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>

username: googlebot
cooked: <p>You’re trying to do classification with a distance based loss. You’d need to output label scores (…,output_size, num_classes), treat them as probabilities (softmax) and use appropriate loss like CrossEntropyLoss. Or do something else, but don’t treat labels as floating point numbers.</p>
<p>NB date embeddings are probably fine, you could also encode them with fourier series coefficients or one-hot encoding, avoiding some backprop.</p>

username: Yasser
cooked: <p>Thank you for your answer,</p>
<p>Actually the label I am trying to predict is a vector of size 720 and each component each represents a “Count”, so it varies from 0 to +infinity, it’s a positive integer value.<br>
Therefore, I think it’s a regression problem or maybe I am totally wrong but I would need more clarification if possible.<br>
I didn’t understand what you meant by this sentence :</p>
<blockquote>
<p>You’d need to output label scores (…,output_size, num_classes)</p>
</blockquote>
<p>I tried to use L1Loss and MSELoss, but both of them give the same problem.<br>
Thank you for the Embedding part, for the moment I will keep it like this and try to solve the problem of overfitting first.</p>

username: googlebot
cooked: <p>Ok, then try using nn.PoissonNLLLoss</p>

username: Yasser
cooked: <p>Hello,</p>
<p>It didn’t work the error wasn’t even decreasing.<br>
Thank you for your help !</p>

username: googlebot
cooked: <p>Well, I’m pretty sure poisson lpdf is the thing to use here. If “overfit” mode doesn’t work, some reasons are:<br>
1)lr is too low, try tuning it or using Adadelta<br>
2)LSTM gates are saturated - 18k inputs is kinda extreme, so you’d need normalization, “compressing” layers and/or other strategies to create good rnn inputs<br>
3)with hidden_size:output_size ratio 1:1 LSTM “memory” is heavily contested by column predictions - some columns may be underfit as a result; to isolate this issue, try optimizing single columns (this issue wouldn’t prevent loss from decreasing though)</p>