FFNN PyTorch vs Keras implementation

yoyo: <p>I am trying to replicate in PyTorch the behavior of a simple FFNN created with Keras.</p>
<p>This is the Keras version:</p>
<blockquote>
<p>initializer = keras.initializers.random_uniform(seed=1)</p>
<pre><code>    model = Sequential([
        Dense(512, activation="relu", input_shape=input_shape, kernel_initializer=initializer),
        Dense(512, activation="relu", kernel_initializer=initializer),
        Dense(num_output, activation="softmax")
    ])

    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=0.01),
        loss="categorical_crossentropy",
        # TODO: change metric
        metrics=[keras.metrics.AUC()]
    )
</code></pre>
</blockquote>
<p>and the model is trained as follows:</p>
<blockquote>
<p>self.model.fit(self.X, self.y, batch_size=batch_size, epochs=epochs, verbose=verbose)</p>
</blockquote>
<p>Now, my PyTorch implementation (it’s my first pytorch NN) is the following:</p>
<blockquote>
<p>class FFNN(nn.Module):<br>
def <strong>init</strong>(self, input_shape, num_classes):</p>
<pre><code>    super(FFNN, self).__init__()

    self.net = nn.Sequential(
        nn.Linear(input_shape[0], 512),
        nn.ReLU(),
        nn.Linear(512, 512),
        nn.ReLU(),
        nn.Linear(512, num_classes),
        nn.Softmax(dim=1))
    self.net.apply(self.init_weights)

def forward(self, X):
    return self.net(X)

def init_weights(self, m):
    if type(m) == nn.Linear:
        m.weight.data.uniform_(-0.05, 0.05)
</code></pre>
</blockquote>
<p>and it is trained as follows:</p>
<blockquote>
<pre><code>    y_idx = get_num_from_1hot(y)

    # convert to tensor
    X_tr = Variable(torch.from_numpy(X).float(), requires_grad=False)
    y_tr = Variable(torch.from_numpy(y_idx), requires_grad=False)

    # Loss and Optimizer
    optimizer = torch.optim.Adam(self.model.parameters(), lr=0.01)
    loss_func = torch.nn.CrossEntropyLoss()

    for i in range(epochs):
        y_pred = self.model(X_tr)

        loss = loss_func(y_pred, y_tr)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
</blockquote>
<p>Unfortunately the Pytorch version seems to get stuck in the training (test accuracy stops at a very specific value for MANY epochs – sometimes until the end of the training).<br>
Can you spot any problem with my implementation?</p>

mathematics: <p>I think you should check pytorch version. using</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)
</code></pre>
<p>latest version as of 2020 is <code>1.6.0</code></p>
<p>That <code>Variable</code> is deprecated in new pytorch version. Instead you could use<br>
<code>autograd_tensor = torch.randn((2, 3, 4), requires_grad=True)</code> , See here <a href="https://pytorch.org/docs/stable/autograd.html#variable-deprecated" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#variable-deprecated</a>,</p>
<p>I think You should go through <a href="https://pytorch.org/tutorials/" rel="nofollow noopener">https://pytorch.org/tutorials/</a>, examples like <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a>  to see it working.</p>

yoyo: <p>I am suing torch 1.4.0.</p>
<p>Thanks for the links, I will definitely follow-up on that.</p>