Problems with implementing a neural network. From Keras to Pytorch

Mirage: <p>Hello everyone! I’m writing this post because I am having trouble implementing a neural network in Pytorch being used to Keras.</p>
<p>Considering the input has dimension (6,3,1) - I am trying to work with time series forecasting, I would like to implement the following network:</p>
<pre><code class="lang-auto">model = Sequential()
model.add(Conv1D(filters=64, kernel_size=2, activation='relu', input_shape=(3, 1)))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(1))
</code></pre>
<p>I tried with:</p>
<pre><code class="lang-auto">MyNet = nn.Sequential(
        nn.Conv1d(3,1,2),
        nn.MaxPool1d(2,stride=None),
        nn.Flatten(start_dim=1),
        nn.Linear(64,50),
        nn.Linear(50, 1)
        )
</code></pre>
<p>but it doesn’t work. I believe the problem is related to Conv1d. Can you please explain me my mistakes?</p>
<p>Thank you!</p>

Nikronic: <p>Hi,</p>
<p>Something that I am sure is wrong is that in your Keras code you have <code>64</code> as number of filters in Conv layer but I cannot see this number in PyTorch equivalent.<br>
I think you might need to replace it with  <code>nn.Conv1d(in_channel=6, out_channel=64, kernel_size=2)</code>.</p>
<p>In PyTorch, we just need to determine in/out channels (number of input and output filters), shapes will be captured by PyTorch itself.</p>
<p>Another issue is that you need to add <code>nn.ReLU()</code> between two linear layers. Activation functions in PyTorch can be used like a separate layer.</p>
<p>Furthermore, I am not sure about your data, but you might need to check which dimension corresponds to features or temporal/spatial dims.<br>
For this, you can follow this thread:<br>
<a href="https://discuss.pytorch.org/t/understanding-convolution-1d-output-and-input/30764">Understanding Convolution 1D output and Input - PyTorch Forums</a></p>
<p>Last 4 replies literally discusses a numerical example.</p>
<p>Bests</p>

Mirage: <p>Hi,</p>
<p>according to <a href="https://discuss.pytorch.org/t/understanding-convolution-1d-output-and-input/30764">Understanding Convolution 1D output and Input</a> my input shape <code>[6, 3, 1]</code> corresponds to <code>[batch_size, in_channel, len]</code>. In addition, <code>out_channel</code> defines the number of the <code>kernels</code>.</p>
<p>As a consequence, MyNet should become:</p>
<pre><code class="lang-auto">MyNet = nn.Sequential(
        nn.Conv1d(in_channels = 3, out_channels = 64, kernel_size = 2),
        nn.MaxPool1d(2),
        nn.Flatten(start_dim=1),
        nn.Linear(64,50),
        nn.ReLU(),
        nn.Linear(50, 1)
        )
</code></pre>
<p>Is it correct?</p>
<p>It doesn’t work, I get the following error: <code>Calculated padded input size per channel: (1). Kernel size: (2). Kernel size can't be greater than actual input size</code></p>
<p>Please, let me know what I am doing wrong!</p>

Nikronic: <p>You only have 6 samples?<br>
What I mean is that you need to define which dimension is corresponding to temporal dim. Based on your explanation, it seems you have only 1 timestamp with 3 features for each sample (=6). In this case, your input data has length of 1, but you have defined kernel size = 2, so it’s not valid.<br>
Can you elaborate your data? Because if <code>in_channel=3</code> and <code>len=1</code>, then you can only define kernel size =1.</p>
<p>In the reference I provided, notice the <code>permute</code> of channels where temporal dimension is moved to last channel.</p>

ptrblck: <p>Besides what <a class="mention" href="/u/nikronic">@Nikronic</a> already explained, I think TF uses the <code>channels_last</code> format by default, so I assume the input shape corresponds to <code>[batch_size=6, seq_len=3, channels=1]</code> and has to be permuted to fit the expected input for <code>nn.Conv1d</code> as <code>[batch_size, channels, seq_len]</code>.</p>
<p>Also, you are still missing another <code>nn.ReLU</code> after the conv layer.</p>

Mirage: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, you are right about the input shape correspondece to <code>[batch_size = 6, seq_len = 3, channels = 1]</code>! <a class="mention" href="/u/nikronic">@Nikronic</a>, yes I only have <code>6 samples</code>. I permuted the input so to get one of dimensions <code>(6,1,3) = [batch_size, channels, seq_len]</code>. Yes, I was missing another <code>nn.ReLU()</code> after the <code>Conv1d layer</code>.</p>
<p>I modified MyNet as following:</p>
<pre><code class="lang-auto">f = nn.Sequential(
        nn.Conv1d(1,64,2),
        nn.ReLU(),
        nn.MaxPool1d(2),
        nn.Flatten(start_dim=1),
        nn.Linear(64,50),
        nn.ReLU(),
        nn.Linear(50, 1)
        )
</code></pre>
<p>again, it does not work. I get the following error: <code>Given groups=1, weight of size [64, 1, 2], expected input[6, 6, 3] to have 1 channels, but got 6 channels instead</code><br>
Please, Can you give me any clue about what it’s wrong?</p>

Nikronic: <aside class="quote no-group" data-username="Mirage" data-post="6" data-topic="100296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mirage/48/30157_2.png" class="avatar"> Mirage:</div>
<blockquote>
<p>input[6, 6, 3]</p>
</blockquote>
</aside>
<p>It says your input data is not <code>[6, 1, 3]</code> as you have mentioned. Your network architecture is correct for this input data. Literally, <code>f</code> is getting an input with shape <code>[6, 6, 3]</code>. Can you show how you use your inputs?</p>

Mirage: <p>I start with <code>X</code> being the input and <code>y</code> being the ground truth:</p>
<pre><code class="lang-auto">print(X.shape)
for i in range(len(X)):
  print(X[i], y[i])

(6, 3, 1)
[[10]
 [20]
 [30]] 40
[[20]
 [30]
 [40]] 50
[[30]
 [40]
 [50]] 60
[[40]
 [50]
 [60]] 70
[[50]
 [60]
 [70]] 80
[[60]
 [70]
 [80]] 90
</code></pre>
<p>where <code>[6,3,1]=[batch_size, seq_len, channels]</code>. Then I permute the dimensions of <code>X</code>:</p>
<pre><code class="lang-auto">x_train = torch.FloatTensor(X).permute(0,2,1) 
print(x_train.shape)
print(x_train)

torch.Size([6, 1, 3])
tensor([[[10., 20., 30.]],

        [[20., 30., 40.]],

        [[30., 40., 50.]],

        [[40., 50., 60.]],

        [[50., 60., 70.]],

        [[60., 70., 80.]]])
</code></pre>
<p>Then:</p>
<pre><code class="lang-auto">train = data.TensorDataset(x_train, y_train)
trainloader = data.DataLoader(train, batch_size=len(x_train), shuffle=False)
</code></pre>
<p>Is there something I’m missing?</p>

Nikronic: <p>Actually, no! everything looks fine to me. I even ran your code and it works fine:</p>
<pre><code class="lang-auto">
x = torch.tensor([[[10., 20., 30.]],

                [[20., 30., 40.]],

                [[30., 40., 50.]],

                [[40., 50., 60.]],

                [[50., 60., 70.]],

                [[60., 70., 80.]]])
y = torch.ones(6, 1)

train = data.TensorDataset(x, y)
trainloader = data.DataLoader(train, batch_size=len(x), shuffle=False)

model = nn.Sequential(
        nn.Conv1d(1,64,2),
        nn.ReLU(),
        nn.MaxPool1d(2),
        nn.Flatten(start_dim=1),
        nn.Linear(64,50),
        nn.ReLU(),
        nn.Linear(50, 1)
        )

for batch in trainloader:
    x_batch, y_batch = batch
    print(x_batch.shape)
    print(model(x_batch).shape)
</code></pre>
<p>You may have missed something? like the way you use <code>x</code> after batching to feed to <code>model</code>? Can you show your train loop?</p>
<p>PS. when you set <code>batch_size=len(your whole data)</code>, you will get only 1 batch which contains a tensor [6, 1, 3]. You need to set <code>batch_size=1</code> if you want <code>6x[1, 1, 3]</code> tensors.</p>

Mirage: <p>When it comes to the train loop, first I define the following:</p>
<pre><code class="lang-auto">device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
</code></pre>
<pre><code class="lang-auto">class Learner(pl.LightningModule):
    def __init__(self, model:nn.Module):
        super().__init__()
        self.model = model
    
    def forward(self, x):
        return self.model(x)
    
    def training_step(self, batch, batch_idx):
        x, y = batch      
        y_hat = self.model(x)   
        loss = nn.MSELoss()(y_hat, y)
        logs = {'train_loss': loss}
        return {'loss': loss, 'log': logs}   
    
    def configure_optimizers(self):
        return torch.optim.Adam(self.model.parameters(), lr=0.005)

    def train_dataloader(self):
        return trainloader
</code></pre>
<p>Then I define the model in the following way:</p>
<pre><code class="lang-auto">f = nn.Sequential(
        nn.Conv1d(1,64,2),
        nn.ReLU(),
        nn.MaxPool1d(2),
        nn.Flatten(start_dim=1),
        nn.Linear(64,50),
        nn.ReLU(),
        nn.Linear(50, 1)
        )

model = NeuralDE(f, sensitivity='adjoint', solver='dopri5').to(device)
</code></pre>
<p>and finally:</p>
<pre><code class="lang-auto">learn = Learner(model)
trainer = pl.Trainer(min_epochs=200, max_epochs=300)
trainer.fit(learn)
</code></pre>

Nikronic: <p>Ok, you are using PyTorch Lightning not PyTorch itself. PyTorch Lightning uses different approach to achieve same goal.</p>
<p>Also, You are feeding your <code>f</code> which so far, I assumed that it is the entire model to another nn module called <code>NeuralDE</code>. This <code>NeuralDE</code> is apparently using different structure for model which I don’t know how it works. For instance, if you just define <code>model = f</code>, then your code should work, I think.</p>

Mirage: <p>I believe the problem is related to NeuralDE bacause I defined <code>model = f</code> as you suggested and it worked! Also, I succeded in implementing a neural network with 2 inputs.</p>
<p>It turns out I have to look deeply into the <code>NeuralDE</code> nn module!</p>
<p>Thank you for your time and patience!</p>

Mirage: <p>I’m getting this error:</p>
<pre><code class="lang-auto">The size of tensor a (200) must match the size of tensor b (3) at non-singleton dimension 1
</code></pre>
<p>What does it mean?</p>

Nikronic: <p>Please print full stack trace for error, and the line that causes it.</p>
<p>But I think it’s related to a mathematical operation which dimension mismatch is happening e.g. matrix multiplication of two matrices where dimensions does not match.</p>

Mirage: <p>The thing is that I am trying to implement a Neural ODE (NODE) in Pytorch. I’m following this example: <a href="https://github.com/DiffEqML/torchdyn/blob/master/tutorials/02_classification.ipynb" rel="noopener nofollow ugc">https://github.com/DiffEqML/torchdyn/blob/master/tutorials/02_classification.ipynb</a><br>
If you know a simple implementantion of NODE, please share it with me!</p>
<p>Regarding the error:</p>
<pre><code class="lang-auto">f = nn.Sequential(
        nn.Conv1d(3,64,kernel_size=2),
        nn.ReLU(),
        nn.MaxPool1d(2),
        nn.Flatten(start_dim=1),
        nn.Linear(576,50),
        nn.ReLU(),
        nn.Linear(50, ph)
        )

model = NeuralDE(f, 
                   solver='rk4',
                   sensitivity='autograd',
                   s_span=torch.linspace(0, 1, 10))

learningRate = 0.01
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learningRate)
</code></pre>
<pre><code class="lang-auto">for epoch in range(180):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 200 == 199:
        #if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  #(epoch + 1, i + 1, running_loss / 2000))
                  (epoch + 1, i + 1, running_loss / 200))
            running_loss = 0.0

print('Finished Training')
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-27-15bbd244efe3&gt; in &lt;module&gt;()
     10 
     11         # forward + backward + optimize
---&gt; 12         outputs = model(inputs)
     13         loss = criterion(outputs, labels)
     14         loss.backward()

6 frames
/usr/local/lib/python3.6/dist-packages/torchdiffeq/_impl/rk_common.py in rk4_alt_step_func(func, t, dt, y, k1)
     98     if k1 is None:
     99         k1 = func(t, y)
--&gt; 100     k2 = func(t + dt * _one_third, y + dt * k1 * _one_third)
    101     k3 = func(t + dt * _two_thirds, y + dt * (k2 - k1 * _one_third))
    102     k4 = func(t + dt, y + dt * (k1 - k2 + k3))

RuntimeError: The size of tensor a (3) must match the size of tensor b (200) at non-singleton dimension 1
</code></pre>
<p>If you know any simple implementation of NODE, please share it. Thank you!</p>