Torch model subtly different than Keras model

BlackBird123: <p>Hello folks,</p>
<p>New to Torch and trying to translate a Keras model to Torch. I’m finding a slight difference in model accuracy that makes me think I haven’t done the model translation properly. Here is the code for the two models:</p>
<p>Keras</p>
<pre><code class="lang-auto">def keras_model(input_tensor_length):
    he_uniform_initializer = he_uniform()
    input_tensor = Input(shape=(1, input_tensor_length))

    dense = Dense(
        units=100, activation="relu", kernel_initializer=he_uniform_initializer
    )(input_tensor)

    dense = Dropout(0.5)(dense)

    dense = Dense(
        units=50, activation="relu", kernel_initializer=he_uniform_initializer
    )(dense)

    dense = Dropout(0.5)(dense)

    dense = Dense(
        units=25, activation="relu", kernel_initializer=he_uniform_initializer
    )(dense)

    predictions = Dense(1, activation="linear")(dense)

    sigm = Dense(1, activation="sigmoid")(predictions)

    model = Model(inputs=input_tensor, outputs=sigm)

    model.compile(
        optimizer=Adam(lr=0.0001), loss="binary_crossentropy", metrics=["mae", "acc"]
    )
</code></pre>
<p>Torch</p>
<pre><code class="lang-auto">class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()

        self.mod = nn.Sequential(
            nn.Linear(input_dim,100),
            nn.Linear(100,50),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(50, 25),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(25, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        y = self.mod(x)
        return y

</code></pre>
<p>Both are trained using BCE on 0/1 targets. Binary Accuracy is computed on both validation sets using the same function here and the resulting accuracies are Kears: 0.697 Torch 0.689:</p>
<pre><code class="lang-auto">def binary_accuracy(self,y_true, y_pred):
        N = sum(y_true == np.round(y_pred))
        return N/len(y_true)
</code></pre>

ptrblck: <p>In your Keras model, you are using a <code>ReLU</code> and <code>Dropout</code> after the first linear layer, which is missing in the PyTorch model.<br>
Also, you are using an additional <code>Dropout</code> layer after the <code>nn.Linear(50, 25)</code> layer in PyTorch, which is not present in Keras and it seems you are using two <code>Dense</code> layers as the output, where the first one is using a “linear” activation function while the latter uses <code>sigmoid</code>.</p>

BlackBird123: <p>Thanks for your response! I really appreciate it.</p>
<p>If I’m understanding you correctly, it should be something like this:</p>
<pre><code class="lang-auto">self.mod = nn.Sequential(
            nn.Linear(input_dim,100),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(100, 50),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(50, 25),
            nn.ReLU(),
            nn.Linear(25, 1),
            nn.Linear(1, 1),
            nn.Sigmoid()
        )
</code></pre>
<p>So from this I am learning that the number of units in the Keras definition is the second number in the Torch layer definition. In essence:</p>
<ul>
<li>Dense(units=100, activation=“relu”, kernel_initializer=he_uniform_initializer)</li>
<li>nn.Linear(input_dim, 100)</li>
<li>nn.ReLU()</li>
</ul>
<p>I’m still not certain how to reat the Input() definition from Keras. Is it a linear layer like I have it?</p>
<ul>
<li>
<pre><code>input_tensor = Input(shape=(1, input_tensor_length))
</code></pre>
</li>
</ul>
<p>Also, do you think the he_uniform_initialization matters?</p>

ptrblck: <aside class="quote no-group" data-username="BlackBird123" data-post="3" data-topic="134239">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/a88e57/48.png" class="avatar"> BlackBird123:</div>
<blockquote>
<p>So from this I am learning that the number of units in the Keras definition is the second number in the Torch layer definition.</p>
</blockquote>
</aside>
<p>Yes, the specified <code>units</code> in <code>Dense</code> are corresponding to the <code>out_features</code> in PyTorch.</p>
<aside class="quote no-group" data-username="BlackBird123" data-post="3" data-topic="134239">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/a88e57/48.png" class="avatar"> BlackBird123:</div>
<blockquote>
<p>I’m still not certain how to reat the Input() definition from Keras. Is it a linear layer like I have it?</p>
</blockquote>
</aside>
<p>You don’t need to specify input layers or any placeholders in PyTorch and can directly pass the input tensors to the model.</p>
<aside class="quote no-group" data-username="BlackBird123" data-post="3" data-topic="134239">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/a88e57/48.png" class="avatar"> BlackBird123:</div>
<blockquote>
<p>Also, do you think the he_uniform_initialization matters?</p>
</blockquote>
</aside>
<p>Yes, initialization can matter but it also depends on the model.</p>

BlackBird123: <p>Thanks so much for your help! I really appreciate it</p>