Feedback on my 1D CNN needed, dimension and no convergence issue

username: Patrick_H
cooked: <p>Hello Torchers,</p>
<p>I have an issue with the data input from the dataloader wrapper and its dimension not adapted to my CNN inputs.</p>
<p>I am using a 1D CNN for an interpolation problem right now. For debugging purpose I have set the input size to 1 node and the CNN must learn the identity function. (is that meaningful btw?, target/label = input for the identity function). I am also training by batches.</p>
<p>Let me tell you first what I am doing before I ask my questions -</p>
<p>The data is constructed with the following custom dataset constructor and dataloader wrapper:</p>
<pre><code class="lang-auto">class custom_dataset(torch.utils.data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, examples, labels):
        'Initialization'
        
        self.labels = labels
        self.examples = examples

  def __len__(self):
        'Denotes the total number of samples'
        return len(self.examples)

  def __getitem__(self, index):
        'Generates one sample of data'
        # Select sample

        # Load data and get label
        X = self.examples [index]
        y = self.labels [index]

        return X, y
</code></pre>
<p>the wrapping happens here, in comments my data shape:</p>
<pre><code class="lang-auto">    #with train_example.shape = torch.Size([57600])
    #and train_label.shape = torch.Size([57600])
    train_example = np.asarray(train_example)
    train_label = np.asarray(train_label)
    trainset = custom_dataset(train_example,train_label)
</code></pre>
<p>the CNN look like this:</p>
<pre><code class="lang-auto">class ConvNet1D(nn.Module):
    def __init__(self,winlength_size, dropout=0.2):
        super(ConvNet1D, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv1d(1, 16, kernel_size=1, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.layer2 = nn.Sequential(
            nn.Conv1d(16, 32, kernel_size=2, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(64, 20)
        self.fc2 = nn.Linear(20, 1)



    def forward(self, x):

        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>finally the training happens in the following snippet:</p>
<pre><code class="lang-auto">'''optimizer is adam'''
'''criterion is MSE loss, reduce = True'''

   for batch_idx, data in enumerate(train_loader):
        inputs = data[0]
        labels = data[1]
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion (outputs, labels)
        loss.backward()
        optimizer.step()

</code></pre>
<p>first issue that came, the dimension : run with those snippets, with a batch_size of 16, the code would give me the following error:</p>
<blockquote>
<p>expected 3-dimensional input for 3-dimensional weights 16, but got 1 dimensional input of size 16</p>
</blockquote>
<p>by adding new dimensions on my input and label data, I could solve the 1D ConvNet input structure issue which must be 3 dimensional, for channel numbers and lengths.<br>
It is done as following:</p>
<pre><code class="lang-auto">train_example = train_example.reshape(train_example[0].shape, 1,1)
train_labels = train_labels.reshape(train_labels[0].shape,1,1)
</code></pre>
<p>now I have the following mistake:</p>
<blockquote>
<p>Using a target size: torch.Size([16,1,1]) that is different to the input size : torch.Size([16,1]). This will likely lead to incorrect results due to error.</p>
</blockquote>
<p>The input dimension somehow reduced to 2, but I cannot change the input dimension because of the conditions on my inputs given by the convnet, so I need to work with the label dimensions:</p>
<pre><code class="lang-auto">train_labels = train_labels.reshape(train_labels[0].shape, 1)
</code></pre>
<p>here I would have a running code, but I don’t have a clear convergence:</p>
<pre><code class="lang-auto">#3 firsts loss by batch:
3_losses= [0.31,0.181,0.143]

#!PSEUDO CODE! The rest:
the_rest = span(random between 0.005 and 0.09)
</code></pre>
<p>so I changed the CNN to</p>
<pre><code class="lang-auto">class ConvNet1D(nn.Module):
    def __init__(self,winlength_size, dropout=0.2):
        super(ConvNet1D, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv1d(1, 16, kernel_size=1, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.layer2 = nn.Sequential(
            nn.Conv1d(16, 32, kernel_size=2, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=1))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(64, 3000)
        self.fc2 = nn.Linear(3000, 150)
        self.fc3 = nn.Linear(150,1)



    def forward(self, x):

        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out

</code></pre>
<p>to get better results.</p>
<p>now my questions:</p>
<ul>
<li>Is it okay for the CNN to have so many nodes in the fully connected layouts?</li>
<li>Is it okay for the CNN to have more than 2 layers in the fully connected part?</li>
<li>Is it okay to use relus functions on the fully connected part of the CNN?</li>
<li>Can we expect better losses for the 1 node input identity CNN ? With the later CNN architecture I get values ranging from 0.0002 and 0.001, but should I not get better losses for my trivial problem?</li>
<li>Is my identity function really that trivial?</li>
<li>why do I have to change the dims of my labels? actually, the real question is, why did the dimension of inputs reduced to 2?</li>
</ul>
<p>It is a long post it will interest someone, thanks in advance for reading me anyway.</p>
<p>Cheers<br>
P.</p>