Converting Tensorflow code to Pytorch help

Tupac: <p>I have this simple tensorflow code block, what is the equivalent in pytorch? I am stuck trying to code it. I have encountered multiple Runtime errors, due to the dimensions.<br>
This is the tensorflow code:</p>
<pre><code class="lang-auto">conv1 = tf.nn.conv1d(x,f1,stride=1,padding="VALID")
conv1 = tf.nn.bias_add(conv1, b1)
conv1 = tf.nn.sigmoid(conv1)

p1 = tf.layers.average_pooling1d(conv1, pool_size=2, strides=2, padding='VALID')

conv2 = tf.nn.conv1d(p1,f2,stride=1,padding="VALID")
conv2 = tf.nn.bias_add(conv2, b2)
conv2 = tf.nn.sigmoid(conv2)
p2 = tf.layers.average_pooling1d(conv2, pool_size=2, strides=2, padding='VALID')

nn = tf.layers.Flatten()(p2)

fc1 = tf.add(tf.matmul(nn, n1), b3)
fc1 = tf.nn.sigmoid(fc1)
out = tf.add(tf.matmul(fc1, n2), b4)
out = tf.nn.softmax(out)
</code></pre>
<p>How can I implement the same in pytorch? The code below is what I tried, but the dimensions are messed up, I think due to the channel inputs.</p>
<pre><code class="lang-auto">class TwoLayerNet(torch.nn.Module):
    def __init__(self):
        super(TwoLayerNet,self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1,3, 3*8, kernel_size=5, stride=1),  
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=0))
        self.conv2 = nn.Sequential(
            nn.Conv1d(3*8, 12, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride = 0))
        #self.drop_out = nn.Dropout()

        self.fc1 = nn.Linear(708, 732) 
        self.fc2 = nn.Linear(732, 4)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out
</code></pre>

albanD: <p>Hi,</p>
<p>You seem to be mixing conv1d and conv2d. Also your conv2d definition has too many arguments I think.</p>

Tupac: <p>Should it only be conv1d? Because I used all conv1d, but there was an error since the input was 4D</p>

albanD: <p>If you input is of size <code>[708, 256, 3]</code>, then you do want conv1d just like in the Tensorflow code.<br>
Also the stride argumnets for the pooling are not the same…</p>

Tupac: <p>Could you please guide on editing the code I posted? I’m really lost…</p>

albanD: <p>The tensorflow code has <code>strides=2</code> while the pytorch version has <code>nn.AvgPool1d(kernel_size=2, stride=0)</code>. So change it to <code>stride=2</code>?</p>

Tupac: <p>Ok thanks, but I’m still confused about the dimensions for the conv1d - what should be the input and output number? And the for the linear part too. I really appreciate your help on this!</p>

albanD: <p>I think you main problem is that Tensorflow input has size <code>[708, 256, 3]</code> with <code>[batch, width, channels]</code>. While pytorch expect <code>[batch, channels, width]</code>.<br>
So you need to be careful to give an input of size <code>[708, 3, 256]</code> to the pytorch model. You should check the doc for these functions to make sure you’re doing the right thing.</p>

Tupac: <p>Thank you so much. I have updated my code, I have a new error:</p>
<pre><code class="lang-auto">class CNN(torch.nn.Module):
    def __init__(self):
        super(CNN,self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv1d(708, 252, kernel_size=5, stride=1),  
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=2))
        self.conv2 = nn.Sequential(
            nn.Conv1d(252, 12, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=2))

        self.fc1 = nn.Linear(61, 12) 
        self.fc2 = nn.Linear(12, 4)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>Error:<br>
—&gt; 24     loss_train = criterion(output_train, ytrain)<br>
ValueError: Expected input batch_size (3) to match target batch_size (708).</p>
<p>These are the shapes. I am comparing them for the Cross Entropy Loss:<br>
ytrain_shape torch.Size([708, 4])<br>
Outputtrain shape torch.Size([3, 12, 4])</p>

albanD: <p>You are missing the operation that correspond to <code>nn = tf.layers.Flatten()(p2)</code>.<br>
You want to add in your <code>forward</code> <code>out = out.view(out.size(0), -1)</code> beween the last convolution and the first fully connected.</p>

Tupac: <aside class="quote no-group" data-username="albanD" data-post="10" data-topic="66123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>out = out.view(out.size(0), -1)</p>
</blockquote>
</aside>
<p>Ok, I have updated the block as below:</p>
<pre><code class="lang-auto">class CNN(torch.nn.Module):
    def __init__(self):
        super(CNN,self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv1d(708, 252, kernel_size=5, stride=1),  
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=2))
        self.conv2 = nn.Sequential(
            nn.Conv1d(252, 12, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=2))

        self.fc1 = nn.Linear(732, 708) 
        self.fc2 = nn.Linear(708, 4)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>Now I get this error for the loss function:<br>
ValueError: Expected input batch_size (3) to match target batch_size (708).</p>
<p>These are the dimensions, how do I ensure y_hat is (708,4)<br>
ytrain_shape: torch.Size([708, 4])<br>
Y_hat shape: torch.Size([3, 4])</p>

albanD: <p>Could you give the full code sample please? I’m not sure what <code>Y_hat</code> is supposed to be here?<br>
It looks like your input does not have the right dimensions as the first conv1d should have 3 input channels not <code>708</code>. You want <code>batch x channels x width</code> for the input.</p>

Tupac: <p>Here is the code (starting from x_train and y_train)</p>
<p>shape of x_train: torch.Size([3, 708, 256])<br>
shape of y_train: torch.Size([708, 4]))</p>
<pre><code class="lang-auto">class CNN(torch.nn.Module):
    def __init__(self):
        super(CNN,self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv1d(708, 252, kernel_size=5, stride=1),  
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=2))
        self.conv2 = nn.Sequential(
            nn.Conv1d(252, 12, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=2))

        self.fc1 = nn.Linear(732, 708) 
        self.fc2 = nn.Linear(708, 4)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)
        return out
</code></pre>
<pre><code class="lang-auto"># defining the model
model = CNN()
model = model.float()
# defining the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.07)
# defining the loss function
criterion = CrossEntropyLoss()

# checking if GPU is available
if torch.cuda.is_available():
    model = model.cuda()
    criterion = criterion.cuda()
    
print(model)
</code></pre>
<pre><code class="lang-auto">def train(epoch):
    model.train()
    tr_loss = 0
    # getting the training set
    x_train, ytr= Variable(X_train), Variable(y_train)
    ytrain = ytr.to(dtype=torch.int64)
    # getting the validation set
    #x_val, y_val = Variable(val_x), Variable(val_y)
    # converting the data into GPU format
    
    print('ytrain_shape',ytrain.shape)

    # clearing the Gradients of the model parameters
    optimizer.zero_grad()
    
    # prediction for training and validation set
    output_train = model(x_train.float())
    #output_train = model(x_train)
    #output_train=output_train.squeeze(1)
    
    print('Y_hat shape',output_train.shape)


    # computing the training and validation loss
    loss_train = criterion(output_train, ytrain)
    

    train_losses.append(loss_train)


    # computing the updated weights of all the model parameters
    loss_train.backward()
    optimizer.step()
    tr_loss = loss_train.item()
    
    return train_losses
</code></pre>
<pre><code class="lang-auto"># defining the number of epochs
n_epochs = 5
# empty list to store training losses
train_losses = []
# empty list to store validation losses
val_losses = []
# training the model
for epoch in range(n_epochs):
    train(epoch)
</code></pre>

albanD: <p>As mentionned above, the shape of x_train is wrong. It should be ([708, 3, 256])</p>

Tupac: <aside class="quote no-group" data-username="albanD" data-post="8" data-topic="66123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>I think you main problem is that Tensorflow input has size <code>[708, 256, 3]</code> with <code>[batch, width, channels]</code> . While pytorch expect <code>[batch, channels, width]</code> .</p>
</blockquote>
</aside>
<p>But then I will get this error:<br>
RuntimeError: Given groups=1, weight of size 252 708 5, expected input[708, 3, 256] to have 708 channels, but got 3 channels instead</p>

albanD: <aside class="quote no-group" data-username="albanD" data-post="12" data-topic="66123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>It looks like your input does not have the right dimensions as the first conv1d should have 3 input channels not <code>708</code> . You want <code>batch x channels x width</code> for the input.</p>
</blockquote>
</aside>
<p>Your first conv parameters are not correct, the input has 3 channels, not 708.</p>

Tupac: <aside class="quote no-group" data-username="albanD" data-post="8" data-topic="66123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>I think you main problem is that Tensorflow input has size <code>[708, 256, 3]</code> with <code>[batch, width, channels]</code> . While pytorch expect <code>[batch, channels, width]</code> .</p>
</blockquote>
</aside>
<p>See here, the thing is in TensorFlow, 708 is not the batch size. I have a 3-D input (3 channels, of 708 rows and 256 columns.</p>
<p>Anyway I have changed as you suggested, but I get this error now</p>
<p>RuntimeError: multi-target not supported at C:\w\1\s\tmp_conda_3.7_104508\conda\conda-bld\pytorch_1572950778684\work\aten\src\THNN/generic/ClassNLLCriterion.c:22<br>
at</p>
<p>—&gt; 25     loss_train = criterion(output_train, ytrain)</p>

albanD: <p>The tensorflow conv1d takes as input <code>batch x width x channel</code> no? So your Tensorflow code is wrong?</p>

Tupac: <p>Ok i got it, i changed as your suggestion. Thanks!</p>