Training issues with basic 1D-CNN

l.u.d.0.v.i.c: <p>Hi everyone,</p>
<p>Here’s my problem. I have a basic 1D-CNN trained for regression purposes (my training data are spectra of length 543).<br>
I do not have errors when training (input shape seems in the right format [N_obs, Channel, Width]), but there is definitely a problem with my model since it cannot overfit even on few data (<strong>5 in this example - 4 in training 1 in test</strong>) and my training loss curve looks weird (it decreases very quicky then reach a plateau and stays there <em>ad vitam aeternam</em>).</p>
<p>I think I need an external point of view to see if there is someting wrong in my code. Thank you for you help.</p>
<p>Here’s my complete code.</p>
<p>My main function:</p>
<pre><code class="lang-auto">    num_epochs = 300
    device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"Using {device} device")

    X_train, X_test, y_train, y_test = train_test_split(
        X_train, y_train, train_size=0.8)
    
    print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
    # output: (4, 543) (1, 543) (4,) (1,)
    
    train_data = TabularDataset(X_train, y_train)
    test_data = TabularDataset(X_test, y_test)

    model = ConvNet_1D()
    optimizer = Adam(model.parameters(), lr=0.001)

    train_dataloader = DataLoader(train_data, batch_size=128, shuffle=True)
    test_dataloader = DataLoader(test_data, batch_size=128, shuffle=True)

    for X, y in test_dataloader:
        print("Shape of X: ", X.shape, X.dtype)
        # Output: Shape of X:  torch.Size([1, 1, 543]) torch.float64
        print("Shape of y: ", y.shape, y.dtype)
        break

    # # Model training
    losses_x: list = []
    losses_y: list = []
    for epoch in range(0, num_epochs):
        print(f"Epoch {epoch + 1}\n----------------------------")
        loss_x = training_function(train_dataloader, model, optimizer, device)
        loss_y = testing_function(test_dataloader, model, device)
        losses_x.append(loss_x)
        losses_y.append(loss_y)
</code></pre>
<p>My CNN architecture:</p>
<pre><code class="lang-auto">from torch import nn

class ConvNet_1D(nn.Module):
    '''
    Defines a 1D-CNN architecture with batch normalization and max pooling layers.
    '''
    def __init__(self, dropout=0):
        super(ConvNet_1D, self).__init__()

        self.ConvNet = nn.Sequential(
            nn.Conv1d(in_channels=1, out_channels=32, kernel_size=3),
            nn.ReLU(),
            nn.BatchNorm1d(32),
            nn.MaxPool1d(kernel_size=2),

            nn.Conv1d(in_channels=32, out_channels=64, kernel_size=3),
            nn.ReLU(),
            nn.BatchNorm1d(64),
            nn.MaxPool1d(kernel_size=2),

            nn.Conv1d(in_channels=64, out_channels=128, kernel_size=3),
            nn.ReLU(),
            nn.BatchNorm1d(128),
            nn.MaxPool1d(kernel_size=2),

            nn.Conv1d(in_channels=128, out_channels=256, kernel_size=3),
            nn.ReLU(),
            nn.BatchNorm1d(256),
            nn.MaxPool1d(kernel_size=2),

            nn.Dropout(dropout),
            nn.Flatten(),
        )

        self.task = nn.Sequential(
            nn.Linear(8192, 100),
            nn.Dropout(dropout),
            nn.ReLU(),
            nn.Linear(100, 1),
        )

    def forward(self, x):
        x = self.ConvNet(x)
        output = self.task(x)

        return output
</code></pre>
<p>My dataloader:</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch.utils.data import Dataset


class TabularDataset(Dataset):
    def __init__(self, train, labels, transform=None):

        self.train = train
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.labels)
    
    def get_data(self):
        return self.train

    def __getitem__(self, idx):
        x = torch.tensor(self.train[idx]).unsqueeze(0)
        y = torch.tensor(self.labels[idx])

    # Apply various transformations to your data here
        if self.transform:
            x, y = self.transform(x), self.transform(y)

        return x, y
</code></pre>
<p>My training function:</p>
<pre><code class="lang-auto">import torch

from xxx.utils import RMSELoss


def training_function(dataloader, model, optimizer, device, scheduler=None):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to(device), y.to(device)

        # Compute the prediction error
        pred = model(X.float())
        loss = RMSELoss(pred, y)
        
        # Reset the gradient to 0 after each epoch
        optimizer.zero_grad()

        # Compute the backpropagation (back propagate the loss and add one step to the optimizer)
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss = loss.item()
            print(f"training loss: {loss:&gt;7f}")
        
        return loss

def testing_function(dataloader, model, device):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    model.eval()
    test_loss, correct = 0, 0

    # set the gradient to 0 (since we do not need to calculate the gradient on the test set)
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to(device), y.to(device)
            pred = model(X.float())
            test_loss += RMSELoss(pred, y).item()
            correct += (pred == y).type(torch.float).sum().item()
    test_loss /= num_batches
    correct /= size
    print(f"Validation loss: {test_loss:&gt;8f} \n")
    return test_loss
</code></pre>

tom: <aside class="quote no-group quote-modified" data-username="l.u.d.0.v.i.c" data-post="1" data-topic="150605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l.u.d.0.v.i.c/48/29648_2.png" class="avatar"> l.u.d.0.v.i.c:</div>
<blockquote>
<p>(it decreases very quicky then reach a plateau and stays there <em>ad vitam eternam</em>).</p>
</blockquote>
</aside>
<p>The standard answer to “the loss doesn’t decrease enough”/“cannot overfit a small amount of data” is “increase capacity”, i.e. use more channels etc.  What happens if you play with the number of channels?</p>
<p>Best regards</p>
<p>Thomas</p>

l.u.d.0.v.i.c: <p>Hi Thomas!</p>
<p>I tried to increase the model complexity (channels x2), but it didn’t improve the training loss. It’s even the opposite. With only 10 data (8 in training and 2 in test) I get a pretty big RMSE on the training set while I get almost 0 with a simple Gradient Boosting model (not optimized).</p>
<p>Here’s my training and test losses:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2b628fdbc34cfd9cbf59a04ef4088f886e284e3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b2b628fdbc34cfd9cbf59a04ef4088f886e284e3" title="Screen Shot 2022-05-02 at 2.34.22 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2b628fdbc34cfd9cbf59a04ef4088f886e284e3_2_667x499.jpeg" alt="Screen Shot 2022-05-02 at 2.34.22 PM" data-base62-sha1="puXguvq1IQYV6r9T5eKFZL3DyLx" width="667" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2b628fdbc34cfd9cbf59a04ef4088f886e284e3_2_667x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2b628fdbc34cfd9cbf59a04ef4088f886e284e3_2_1000x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2b628fdbc34cfd9cbf59a04ef4088f886e284e3.jpeg 2x" data-dominant-color="FAFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-02 at 2.34.22 PM</span><span class="informations">1212×908 76 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <p>Three things I would look at (from “duh”-moments I have had over the years) are</p>
<ul>
<li>are dimensions all used right, e.g. not using the batch dimension as features or somesuch (not as easy with CNNs as with RNNs, but worth double checking)?</li>
<li>do gradients vanish? (you have batch norm, but still…)</li>
<li>is the range the model produce actually match the range of your targets (should be OK with ReLU + Linear at the end, mostly a problem with Tanh and friends).</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

InnovArul: <aside class="quote no-group" data-username="l.u.d.0.v.i.c" data-post="1" data-topic="150605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l.u.d.0.v.i.c/48/29648_2.png" class="avatar"> l.u.d.0.v.i.c:</div>
<blockquote>
<pre><code class="lang-auto">class TabularDataset(Dataset):
    def __init__(self, train, labels, transform=None):
</code></pre>
</blockquote>
</aside>
<aside class="quote no-group" data-username="l.u.d.0.v.i.c" data-post="1" data-topic="150605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l.u.d.0.v.i.c/48/29648_2.png" class="avatar"> l.u.d.0.v.i.c:</div>
<blockquote>
<pre><code class="lang-auto">    train_data = TabularDataset(X_train, y_train)
    test_data = TabularDataset(X_test, y_test)
</code></pre>
</blockquote>
</aside>
<p>The data is not being normalized. Is it intended?</p>

l.u.d.0.v.i.c: <p>Thanks Arul for your answer.<br>
I do have a normalization stage but it’s not in my TabularDataset class (but I could move it to this class indeed).</p>

InnovArul: <aside class="quote no-group" data-username="l.u.d.0.v.i.c" data-post="1" data-topic="150605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l.u.d.0.v.i.c/48/29648_2.png" class="avatar"> l.u.d.0.v.i.c:</div>
<blockquote>
<p>my training loss curve looks weird (it decreases very quicky then reach a plateau and stays there</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="l.u.d.0.v.i.c" data-post="1" data-topic="150605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l.u.d.0.v.i.c/48/29648_2.png" class="avatar"> l.u.d.0.v.i.c:</div>
<blockquote>
<p><code>optimizer = Adam(model.parameters(), lr=0.001)</code></p>
</blockquote>
</aside>
<p>If possible, try the following things and observe the behavior:</p>
<ol>
<li>Play around with the learning rate (Specifically, try to reduce the learning rate and see if it helps)</li>
<li>Try adding <code>weight_decay</code> parameter</li>
<li>Replace <code>Adam</code> with <code>SGD</code> and see what happens</li>
</ol>

l.u.d.0.v.i.c: <p>My model receive in input a shape [128, 1, 543] ([batch_size, N_channel, N_features]), so it should be ok for this part.</p>
<p>For the gradient vanishing, it’s indeed on my list of things to check (but I’m still not sure how to verify that).</p>
<p>My model should predict values from 0 to a certain value (it depends on the target but not more that 20 000), so Linear at the end should be ok.</p>

l.u.d.0.v.i.c: <p>Thanks Arul.</p>
<ol>
<li>If I decrease or increase the learning rate, the performance is a bit different but it is still underfitting.</li>
<li>With 0.05 weight_decay I have similar results.</li>
<li>With SGD I have a flat training and test loss (if I increase the learning rate, I think I get exploding gradient with nan in losses).</li>
</ol>

InnovArul: <aside class="quote no-group" data-username="l.u.d.0.v.i.c" data-post="9" data-topic="150605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l.u.d.0.v.i.c/48/29648_2.png" class="avatar"> l.u.d.0.v.i.c:</div>
<blockquote>
<p>underfitting</p>
</blockquote>
</aside>
<p>When it comes to underfitting, <a class="mention" href="/u/tom">@tom</a>’s suggestion to increase the model capacity is in the right direction.</p>
<aside class="quote no-group" data-username="l.u.d.0.v.i.c" data-post="8" data-topic="150605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l.u.d.0.v.i.c/48/29648_2.png" class="avatar"> l.u.d.0.v.i.c:</div>
<blockquote>
<p>it depends on the target but not more that 20 000</p>
</blockquote>
</aside>
<p><code>2 x 10^4</code> is quite a large value for a target. I would try to normalize the target as well to be in a smaller range. (preferably, <code>[-1, 1]</code>)</p>

l.u.d.0.v.i.c: <p>My model starts to have a normal behaviour in terms of train/validation loss with normalized labels. Thanks.</p>