Basic MNIST Keras model to Pytorch implementation

username: jarvico
cooked: <p>Hi all,</p>
<p>I need to implement below model structure defined in Keras using pytorch.</p>
<pre><code>    # MNIST model
    layers = [
        Conv2D(64, (3, 3), padding='valid', input_shape=(28, 28, 1)),
        Activation('relu'),
        Conv2D(64, (3, 3)),
        Activation('relu'),
        MaxPooling2D(pool_size=(2, 2)),
        Dropout(0.5),
        Flatten(),
        Dense(128),
        Activation('relu'),
        Dropout(0.5),
        Dense(10),
        Activation('softmax')
    ]
</code></pre>
<p>I tried below model in Pytorch, but I would like to double check if I did it correctly:</p>
<pre><code class="lang-auto">class LeNet_dropout(nn.Module):
    def __init__(self):
        super(LeNet_dropout,self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=64, kernel_size=3, padding=0, bias=True)
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=0, bias=True)
        self.dropout = nn.Dropout(0.5)
        self.fc1 = nn.Linear(12*12*64, 128)
        self.fc2 = nn.Linear(128, 10)


    def last_hidden_layer_output(self, X):
        X = F.relu(self.conv1(X))
        X = self.dropout(F.max_pool2d(F.relu(self.conv2(X)),2))
        X = X.view(-1, 12*12*64)
        X = self.dropout(F.relu(self.fc1(X)))
        return X

    def forward(self, X):
        X = F.relu(self.conv1(X))
        X = self.dropout(F.max_pool2d(F.relu(self.conv2(X)),2))
        X = X.view(-1, 12*12*64)
        X = self.dropout(F.relu(self.fc1(X)))
        X = torch.softmax(F.relu(self.fc2(X)), dim=1)
        return X
</code></pre>
<p>Would you please comment and correct me if I did anything wrong?</p>
<p>And I am not sure which loss function to use to train this model. Because there is a softmax in output. I would appreciate if you can also guide how to train exactly? I think I cannot use nn.CrossEntropyLoss as below:</p>
<p>output = Model(data)<br>
loss = nn.CrossEntropyLoss(output,target)</p>
<p>And last thing, I will also need to use the output of to last hidden layer activation. Thats why I also include a method called last_hidden_layer_output. Is there any other way to handle this?</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> can be used for a multi-class classification and expects raw logits as the model output so you should remove the last <code>torch.softmax</code> activation.<br>
Also, remove the last <code>F.relu</code> and return the output of <code>self.fc2(x)</code> directly.</p>
<aside class="quote no-group" data-username="jarvico" data-post="1" data-topic="102664">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>And last thing, I will also need to use the output of to last hidden layer activation. Thats why I also include a method called last_hidden_layer_output. Is there any other way to handle this?</p>
</blockquote>
</aside>
<p>You could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> or you could keep this method but reuse it in <code>forward</code> to avoid duplicated code:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, x):
    x = self.last_hidden_layer_output(x)
    x = self.fc2(x)
    return x
</code></pre>

username: jarvico
cooked: <p>Thank you ptrblck.</p>
<p>First of all , I needed to use softmax at last layer.</p>
<p>The reason I used F.relu at the last layer is because  when I calculate loss as below I get nan values after some epoch.</p>
<pre><code class="lang-auto">toutput = model(data)
loss = F.nll_loss(torch.log(output), target)
</code></pre>
<p>So, I used Relu at last layer and changes loss calculate as below:</p>
<pre><code class="lang-auto">eps = 1e-7
output = model(data)
loss = F.nll_loss(torch.log(output+eps), target)
</code></pre>
<p>If I remove Relu from last layer, how can I guarantee the term output in torch.log become non negative?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jarvico" data-post="3" data-topic="102664">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>First of all , I needed to use softmax at last layer.</p>
</blockquote>
</aside>
<p>Why would that be the case?</p>
<aside class="quote no-group" data-username="jarvico" data-post="3" data-topic="102664">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>If I remove Relu from last layer, how can I guarantee the term output in torch.log become non negative?</p>
</blockquote>
</aside>
<p><code>F.nll_loss</code> or <code>nn.NLLLoss</code> expects log probabilities as the model output, so you should use:</p>
<pre data-code-wrap="python"><code class="lang-python">output = F.log_softmax(x, dim=1)
loss = F.nll_loss(output, target)
</code></pre>
<p>while <code>nn.CrossEntropyLoss</code> expects raw logits and will apply <code>F.log_softmax</code> + <code>F.nll_loss</code> internally.</p>

username: jarvico
cooked: <p>Hi,</p>
<p>Thank you ptrblck.</p>
<p>I changed my model to:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=64, kernel_size=3, padding=0, bias=True)
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=0, bias=True)
        self.dropout = nn.Dropout(0.5)
        self.fc1 = nn.Linear(12*12*64, 128)
        self.fc2 = nn.Linear(128, 10)

    def last_hidden_layer_output(self, X):
        X = F.relu(self.conv1(X))
        X = self.dropout(F.max_pool2d(F.relu(self.conv2(X)),2))
        X = X.view(-1, 12*12*64)
        X = self.dropout(F.relu(self.fc1(X)))
        return X

    def forward(self, X):
        X = self.last_hidden_layer_output(X)
        X = self.fc2(X)
        return X
</code></pre>
<p>I calculate loss like this:</p>
<pre><code class="lang-auto">        output = model(data)
        loss = F.nll_loss(F.log_softmax(output, dim=1), target)
</code></pre>
<p>I got predictions like this:</p>
<pre><code class="lang-auto">predictions = F.softmax(model(data), dim=1).data.argmax(1, keepdim=True)
</code></pre>
<p>I got last hidden layer output like this:</p>
<pre><code class="lang-auto">model.last_hidden_layer(data).data
</code></pre>

username: ptrblck
cooked: <p>Your code looks generally good. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Some small suggestions:</p>
<ul>
<li>Don’t use the <code>.data</code> attribute, as it could yield unwanted side effects</li>
<li>you can get the predictions directly via <code>torch.argmax(output, dim=1)</code> without applying the <code>softmax</code>, since the max. logit will also have the max. probability. However, you can of course apply the <code>softmax</code>, if you want to see the probabilities (just don’t pass it to the criterion to calculate the loss).</li>
</ul>