CNN architecture for short time series data

username: bibekx
cooked: <p>I want to build a CNN for classifying time series data with high accuracy. The data has been windowed into chunks of 12 time steps, it looks like this:</p>
<pre><code class="lang-auto">       [0.        , 0.00167084, 0.00568087, ..., 0.60016708, 0.55238095,
        0.68421053],
       [0.00167084, 0.00568087, 0.146533  , ..., 0.55238095, 0.68421053,
        0.6566416 ],
</code></pre>
<p>Consider my use of CNN for such data an experiment.</p>
<p>Here are 2 architectures that I have tried. But they have poor accuracy.</p>
<p><strong><span class="hashtag-raw">#Architecture</span> 1:</strong></p>
<p>init:</p>
<blockquote>
<pre><code>    self.conv1 = nn.Conv1d(1,3, kernel_size = 2, stride = 1, padding =1)
    self.bn1 = nn.BatchNorm1d(3)
    self.maxpool1 = nn.MaxPool1d(kernel_size=2, stride=2, padding=1)
    self.dropout1 = nn.Dropout(0.3)
    
    self.conv2 = nn.Conv1d(3, 5, kernel_size=2, stride=2, padding=1)
    self.bn2 = nn.BatchNorm1d(5)
    self.dropout2 = nn.Dropout(0.3)
    
    self.conv3 = nn.Conv1d(5, 5, kernel_size=2, stride=1, padding=1)
    self.bn3 = nn.BatchNorm1d(5)
    self.dropout3 = nn.Dropout(0.3)
    
    self.fc = nn.Linear(5, num_classes)
</code></pre>
</blockquote>
<p>forward:</p>
<blockquote>
<pre><code>    x = self.conv1(x)
    x = self.bn1(x)
    x = F.relu(x)

    x = self.maxpool1(x)
    x = self.dropout1(x)

    x = self.conv2(x)
    x = self.bn2(x)
    x = F.relu(x)

    x = self.dropout2(x)

    x = self.conv3(x)
    x = self.bn3(x)
    x = F.relu(x)

    x = self.dropout3(x)
    x = torch.mean(x, dim = 2)
    x = self.fc(x)
</code></pre>
</blockquote>
<p><strong><span class="hashtag-raw">#Architecture</span> 2: similar to PyTorch MNIST example</strong></p>
<p>init:</p>
<blockquote>
<pre><code>    self.conv1 = nn.Conv1d(1,3, kernel_size = 2, stride = 1, padding =1)
    self.conv2 = nn.Conv1d(3, 5, kernel_size=2, stride=2, padding=1)
    self.maxpool1 = nn.MaxPool1d(kernel_size=2, stride=2, padding=1)
    self.dropout1 = nn.Dropout(0.3)
    self.dropout2 = nn.Dropout(0.3)
    
    self.fc1= nn.Linear(20, 12)
    self.fc2= nn.Linear(12, num_classes)
</code></pre>
</blockquote>
<p>forward:</p>
<blockquote>
<pre><code>    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.relu(x)
    x = self.maxpool1(x)
    x = self.dropout1(x)
    x = torch.flatten(x, 1)
    x = self.fc1(x)
    x = F.relu(x)
    x = self.dropout2(x)
    x = self.fc2(x) 
</code></pre>
</blockquote>
<p>Could anyone suggest what I am doing fundamentally wrong here?<br>
P.S. I want to classify them into 3 classes</p>

username: ptrblck
cooked: <p>I can’t see any fundamental flaws in your models.<br>
You could e.g. apply the relu before the batchnorm layers in your first model, but this might not change too much.</p>
<p>I would recommend to try to overfit a small dataset, e.g. just 10 samples, and make sure your model is able to do so by playing around with the hyperparameters.<br>
If that’s not working, there might be another issue in your training routine, such as forgetting to zero out the gradients.</p>

username: bibekx
cooked: <p>I will give that a try</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  What are your thoughts on using batch norm as well as dropout? I read that batch norm acts like a regularizer so that we dont need dropout.</p>

username: ptrblck
cooked: <p>I’ve read the same a couple of times, but would recommend to run a few experiments to make sure this claim isn’t only valid for some mentioned use cases.</p>