RNN on a Spectrogram - Loss Isn't Going Down

jhan6501: <p>I’m having issues training an RNN on some spectograms on 1 second audio clips. The spectrograms are of shape 271 * 781 * 4, with the 781 being the width (So the time stamp). I turned this to 271 * 780 * 4 so I can divide each data into groups of 4 to feed into the RNN. Currently, I’m having issues getting loss to go down. I’ve debugged an issue where my data wasn’t being broken into groups of 4 properly (it wasn’t being divided by time step), I’ve fixed that so I’m certain that is no longer the issue.</p>
<p>I’m thinking that there are probably issues with my RNN architecture code and my forward, but I"m not too sure how to figure it out. I’ve googled around and haven’t gotten too far (my RNN knowledge in the practical sense isn’t great, but I understand the idea of feeding the hidden output from the previous layer into the next layer in conjunction with the next ‘section’ of the data).</p>
<p>Here is my Model Architecture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/3719e9cda7780a3f02f016764a78f5d4d060c9ab.png" data-download-href="https://discuss.pytorch.org/uploads/default/3719e9cda7780a3f02f016764a78f5d4d060c9ab" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/3719e9cda7780a3f02f016764a78f5d4d060c9ab_2_670x499.png" alt="image" data-base62-sha1="7RrMlKu0AcYk2RAth61Cdx51S9Z" width="670" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/3719e9cda7780a3f02f016764a78f5d4d060c9ab_2_670x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/3719e9cda7780a3f02f016764a78f5d4d060c9ab_2_1005x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/7/3719e9cda7780a3f02f016764a78f5d4d060c9ab.png 2x" data-dominant-color="F2F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1112×829 92.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I made this account so I can only upload 1 image, will try to upload the other code in the comments</p>

jhan6501: <p>Hyperparameters<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92626ea3ce2d6cabb7053cbfb7643a5619e54f70.png" data-download-href="https://discuss.pytorch.org/uploads/default/92626ea3ce2d6cabb7053cbfb7643a5619e54f70" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92626ea3ce2d6cabb7053cbfb7643a5619e54f70_2_689x266.png" alt="image" data-base62-sha1="kSYBeZjYh07xGMWUVJHUstBqjGo" width="689" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92626ea3ce2d6cabb7053cbfb7643a5619e54f70_2_689x266.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92626ea3ce2d6cabb7053cbfb7643a5619e54f70_2_1033x399.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/2/92626ea3ce2d6cabb7053cbfb7643a5619e54f70.png 2x" data-dominant-color="F2F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1194×462 46.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jhan6501: <p>Training Code<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef37e27167b6064eec08bb8432cb819abf894a5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef37e27167b6064eec08bb8432cb819abf894a5e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef37e27167b6064eec08bb8432cb819abf894a5e_2_564x500.png" alt="image" data-base62-sha1="y8dQxRlMuGO456As5VYXm0xTBLg" width="564" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef37e27167b6064eec08bb8432cb819abf894a5e_2_564x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef37e27167b6064eec08bb8432cb819abf894a5e_2_846x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef37e27167b6064eec08bb8432cb819abf894a5e.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1022×906 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nateanl: <p>Hi <a class="mention" href="/u/jhan6501">@jhan6501</a>, I’m not sure how your data looks like. Maybe you can try to add a <code>sigmoid</code> layer after <code>fc</code>. This is a common setup in binary classification task.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits, so the <code>sigmoid</code> usage of an output of <code>[batch_size, 8]</code> would be wrong.</p>
<p><a class="mention" href="/u/jhan6501">@jhan6501</a> I would generally recommend to try to overfit a small dataset first (e.g. just 10 samples) and make sure your model is able to do so. Once this works, you could try to scale up the use case again.</p>
<p>Also, remove the <code>Variable</code> wrappers, since they are deprecated since PyTorch <code>0.4</code>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>