Convolution 1d results: not sure

abodh_ltd: <p>Hi all,</p>
<p>Here are my results after performing 1d convolution. I have the following CNN architecture for a dataset sized [batch, 1, 402].</p>
<pre data-code-wrap="python"><code class="lang-python">class Simple1DCNN(torch.nn.Module):
    def __init__(self):
        super(Simple1DCNN, self).__init__()
        self.layer1 = torch.nn.Conv1d(in_channels = 1, out_channels = 10, kernel_size = 3)
        self.act = torch.nn.ReLU()
        self.layer2 = torch.nn.Conv1d(in_channels = 10, out_channels = 20, kernel_size = 3)
        self.fc1 = torch.nn.Linear(20*398, 1000)
        self.fc2 = torch.nn.Linear(1000, 50)
        self.fc3 = torch.nn.Linear(50, 1)
        self.conv2_drop = torch.nn.Dropout(0.5)
    
    def forward(self, x):
        x = self.act(self.layer1(x))
        x = self.act(self.conv2_drop(self.layer2(x)))
        x = x.view(-1, x.shape[1] * x.shape[-1])
        x = torch.tanh(self.fc1(x))
        x = torch.tanh(self.fc2(x))
        x = self.fc3(x)      # collecting the output of linear layer
        return x

</code></pre>
<p>I have used MSE loss as a loss metric. Does the plots look okay? Since the ouputs are float value (not labels) are the bumps on the validation loss justifiable?</p>
<p>P.S. the accuracy is just for my sake. I have considered the output within 10% of the actual output value to be accurate. How do you think I can minimize the fluctuations of validation accuracy?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e2cf0822af8cbf2b7c4e62ac16a3a875165f753.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e2cf0822af8cbf2b7c4e62ac16a3a875165f753" title="cb950d6b557b8eef73855e9b9ea14c4f5c09b17a_2_1380x952"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e2cf0822af8cbf2b7c4e62ac16a3a875165f753_2_690x476.png" alt="cb950d6b557b8eef73855e9b9ea14c4f5c09b17a_2_1380x952" data-base62-sha1="khK59R0MeHs0CMqxdYzfBouYl9h" width="690" height="476" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e2cf0822af8cbf2b7c4e62ac16a3a875165f753_2_690x476.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e2cf0822af8cbf2b7c4e62ac16a3a875165f753_2_1035x714.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e2cf0822af8cbf2b7c4e62ac16a3a875165f753_2_1380x952.png 2x" data-dominant-color="F7F5F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cb950d6b557b8eef73855e9b9ea14c4f5c09b17a_2_1380x952</span><span class="informations">1380×952 36.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aad11a08f8b15e34b78127840a3c6e77e6f010b0.png" data-download-href="https://discuss.pytorch.org/uploads/default/aad11a08f8b15e34b78127840a3c6e77e6f010b0" title="417849adf011565af4e3336df152a8df9b00b57b_2_1380x942"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aad11a08f8b15e34b78127840a3c6e77e6f010b0_2_690x471.png" alt="417849adf011565af4e3336df152a8df9b00b57b_2_1380x942" data-base62-sha1="on7aqWSfiZ7f4OJ87siI2AJ9eRq" width="690" height="471" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aad11a08f8b15e34b78127840a3c6e77e6f010b0_2_690x471.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aad11a08f8b15e34b78127840a3c6e77e6f010b0_2_1035x706.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aad11a08f8b15e34b78127840a3c6e77e6f010b0_2_1380x942.png 2x" data-dominant-color="F3F1F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">417849adf011565af4e3336df152a8df9b00b57b_2_1380x942</span><span class="informations">1380×942 52.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/173dff5317d427bc95928ffbf490d9f0f613d54b.png" data-download-href="https://discuss.pytorch.org/uploads/default/173dff5317d427bc95928ffbf490d9f0f613d54b" title="5b36b6e4506a9b3ec2c00192a56eadb9ec15ed23_2_1340x1000"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/173dff5317d427bc95928ffbf490d9f0f613d54b_2_670x500.png" alt="5b36b6e4506a9b3ec2c00192a56eadb9ec15ed23_2_1340x1000" data-base62-sha1="3jBOfLXXGbx5aXgeEuIY4vtKNx9" width="670" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/173dff5317d427bc95928ffbf490d9f0f613d54b_2_670x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/173dff5317d427bc95928ffbf490d9f0f613d54b_2_1005x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/173dff5317d427bc95928ffbf490d9f0f613d54b_2_1340x1000.png 2x" data-dominant-color="E1DDD7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">5b36b6e4506a9b3ec2c00192a56eadb9ec15ed23_2_1340x1000</span><span class="informations">1340×1000 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For a test set,</p>
<p>The trained model outputs the following:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f6fe76c26947299b7f3f604d5b875faace72de9.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f6fe76c26947299b7f3f604d5b875faace72de9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f6fe76c26947299b7f3f604d5b875faace72de9.png" alt="image" data-base62-sha1="ibmoiEhSjakMQi4xxV2uxi7MKCd" width="690" height="150" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1042×227 12.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How to improve the test accuracy?</p>

ptrblck: <p>How large is your validation set and is the accuracy drop reproducible in this particular epoch?<br>
To further improve your model, you could try to plot the <code>log(loss)</code> and see, if the training loss is decreasing, as it’s a bit hard to tell from the current plot.<br>
If that’s not the case, you could add more aggressive data augmentation and increase the model capacity (more or bigger layers). Once your model starts to overfit, add regularization.</p>

abodh_ltd: <p><em><strong>Don’t mind the edits in this thread.</strong></em></p>
<p>The length of my dataset is 1700 and I use 80% (1360) for training and 20% (340) for validation. To clarify you, my dataset looks something like the sketch below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/651365fa5e6e273da49bffc321b65068b81961a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/651365fa5e6e273da49bffc321b65068b81961a7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/651365fa5e6e273da49bffc321b65068b81961a7_2_690x427.png" alt="image" data-base62-sha1="eq9MGF3oZcNm7dgM07CIkIOalgz" width="690" height="427" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/651365fa5e6e273da49bffc321b65068b81961a7_2_690x427.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/651365fa5e6e273da49bffc321b65068b81961a7_2_1035x640.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/651365fa5e6e273da49bffc321b65068b81961a7.png 2x" data-dominant-color="262323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1234×764 87.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Each snapshot (i.e. each of the waveform from 0 - t) represents a value M. The way I am training is, feeding each of the data points of the waveform as input (for e.g. for green signal, the input will be t points  from its waveform and the network will have to provide M = 3 as an output). P.S. some of the wave forms can be very close for values such as M = 9 and M = 9.5. How do you think I can use data augmentation techniques here? I would highly appreciate your suggestion on this.</p>
<p>Meanwhile, I lost the seed since I am using jupyter notebook. Hence, I was not able to reproduce the exact same results. Sorry about that. I again ran the simulation  for different seed value.</p>
<p>Here are the log(losses) and accuracy. What do you think about these plots? Is log(loss) the right thing to look even if I am using torch.nn.MSE() as my loss metric? I am not sure if I plotted the loss loss correctly. I just used np.log(losses) where I got losses from MSE.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a311de812f1042c397a64d47aa2688ffef8148dd.png" alt="image" data-base62-sha1="ngA6XdqnqMM6iazrB090ODFXa1v" width="407" height="274"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/107e811ad96409592b1e268cc284597fb9c5fe83.png" alt="image" data-base62-sha1="2lUFLZeda2ipin5HRyCABoMprDJ" width="407" height="274"></p>

abodh_ltd: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any insights ?</p>

ptrblck: <aside class="quote no-group" data-username="abodh_ltd" data-post="3" data-topic="77229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abodh_ltd/48/22637_2.png" class="avatar"> abodh_ltd:</div>
<blockquote>
<p>How do you think I can use data augmentation techniques here?</p>
</blockquote>
</aside>
<p>That’s a tricky question, as I’m not familiar with the data.<br>
You would have to narrow down certain properties if a specific class, e.g. <code>M=9</code>, and compare how the waveforms differ for this particular class.<br>
Are they shifted or scaled slightly? Do they have more “noise” in the signal?<br>
Note that you would have to avoid such aggressive augmentation techniques that your data sample might be pushed to another class.</p>
<p>Also, knowledge about the underlying physical process might be useful.<br>
I.e. how was the data created and what kind of “noise” might the recording setup experience?</p>
<aside class="quote no-group" data-username="abodh_ltd" data-post="3" data-topic="77229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abodh_ltd/48/22637_2.png" class="avatar"> abodh_ltd:</div>
<blockquote>
<p>What do you think about these plots? Is log(loss) the right thing to look even if I am using torch.nn.MSE() as my loss metric? I am not sure if I plotted the loss loss correctly. I just used np.log(losses) where I got losses from MSE.</p>
</blockquote>
</aside>
<p>Yes, the <code>log</code> should be fine for visualization purposes and helps a bit in seeming the decreasing behavior for small (close to 0) values.</p>
<p>The loss curves look generally good and it seems your model is still learning.<br>
Do you see some overfitting after a while or a static loss?</p>

abodh_ltd: <p>Thank you for your response.</p>
<p>The data was created using a linearized model of a system. Generally for higher values of M i.e. M = 8,9 or 10 (could also be continuous values like 8.8, 10.2, 9.7) the curve gets really close. Since the signals are measured, they already contain some noise (assumed to be gaussian).</p>
<p>For now with 150 epochs of training, I am getting an accuracy of about 96% which seems good I guess. The model does not overfit with higher epochs. i believe, the problem is solved for now. Thank you for your suggestions and tips (specially the observation of log (loss) for small values of losses) . This will come in very handy for my future works.</p>