Pytorch equivalent of Keras 'categorical_crossentropy' loss function and fixing my Pytorch model code

Bydoless: <p>I’m trying to convert CNN model code from Keras with a Tensorflow backend to Pytorch.</p>
<p>Problem is that I can’t seem to find the equivalent of Keras’ ‘categorical crossentrophy’ function:</p>
<p>model.compile(loss=‘categorical_crossentropy’, optimizer=‘adam’, metrics=[‘accuracy’])</p>
<p>The closest I can find is this:</p>
<p>self._criterion = nn.CrossEntropyLoss()<br>
self._optimizer = optim.Adam(self._model.parameters(), eps=1e-07)<br>
…<br>
loss = self._criterion(outputs, primary_indexes)<br>
loss.backward()</p>
<p>But it doesn’t function similarly and as well as the original Keras code. It takes twice as many epochs to end on the original dataset and doesn’t work as well, and in my larger datasets the loss and accuracy goes from around ~15-20% at the first epoch to around 4% when training ends.</p>
<p>Whereas the Keras version goes from ~15-20% to around ~40-55% when training ends.</p>
<p>Here’s the original Keras model:</p>
<blockquote>
<p>model = models.Sequential()<br>
model.add(Reshape(in_shp+[1], input_shape=in_shp))<br>
model.add(ZeroPadding2D((0,2)))<br>
model.add(Conv2D(64, (1,4), activation=“relu”))<br>
model.add(Dropout(dr))<br>
model.add(ZeroPadding2D((0,2)))<br>
model.add(Conv2D(64, (2,4), activation=“relu”))<br>
model.add(Dropout(dr))<br>
model.add(Conv2D(128, (1,8), activation=“relu”))<br>
model.add(Dropout(dr))<br>
model.add(Conv2D(128, (1,8), activation=“relu”))<br>
model.add(Dropout(dr))<br>
model.add(Flatten())<br>
model.add(Dense(256, activation=‘relu’))<br>
model.add(Dropout(dr))<br>
model.add(Dense(len(classes), activation=‘softmax’))<br>
model.add(Reshape([len(classes)]))<br>
model.compile(loss=‘categorical_crossentropy’, optimizer=‘adam’, metrics=[‘accuracy’])<br>
model.summary()</p>
</blockquote>
<blockquote>
<p>Layer (type)                   Output Shape              Param #</p>
<p>reshape_1 (Reshape)            (None, 2, 128, 1)         0</p>
<hr>
<p>zero_padding2d_1 (ZeroPadding) (None, 2, 132, 1)         0</p>
<hr>
<p>conv2d_1 (Conv2D)              (None, 2, 129, 64)        320</p>
<hr>
<p>dropout_1 (Dropout)            (None, 2, 129, 64)        0</p>
<hr>
<p>zero_padding2d_2 (ZeroPadding) (None, 2, 133, 64)        0</p>
<hr>
<p>conv2d_2 (Conv2D)              (None, 1, 130, 64)        32832</p>
<hr>
<p>dropout_2 (Dropout)            (None, 1, 130, 64)        0</p>
<hr>
<p>conv2d_3 (Conv2D)              (None, 1, 123, 128)       65664</p>
<hr>
<p>dropout_3 (Dropout)            (None, 1, 123, 128)       0</p>
<hr>
<p>conv2d_4 (Conv2D)              (None, 1, 116, 128)       131200</p>
<hr>
<p>dropout_4 (Dropout)            (None, 1, 116, 128)       0</p>
<hr>
<p>flatten_1 (Flatten)            (None, 14848)             0</p>
<hr>
<p>dense1 (Dense)                 (None, 256)               3801344</p>
<hr>
<p>dropout_5 (Dropout)            (None, 256)               0</p>
<hr>
<p>dense2 (Dense)                 (None, 11)                2827</p>
<hr>
<p>reshape_2 (Reshape)            (None, 11)                0</p>
</blockquote>
<p>My Pytorch model code looks like this:</p>
<pre><code>def __init__(self, classes, dr=0.5, bias=False):
	super(My_Model, self).__init__()

	self.pad = nn.ConstantPad2d((2, 2, 0, 0), 0)
	self.dropout = nn.Dropout(dr)
	self.conv1 = nn.Conv2d(  1,  64, (1,4), bias=bias)
	self.conv2 = nn.Conv2d( 64,  64, (2,4), bias=bias)
	self.conv3 = nn.Conv2d( 64, 128, (1,8), bias=bias)
	self.conv4 = nn.Conv2d(128, 128, (1,8), bias=bias)
	self.linear1 = nn.Linear(self.linear1_input_size,256, bias=bias)
	self.linear2 = nn.Linear(256,classes, bias=bias)

def forward(self, x):
	x = self.pad(x)
	x = F.relu(self.conv1(x))
	x = self.dropout(x)
	x = self.pad(x)
	x = F.relu(self.conv2(x))
	x = self.dropout(x)
	x = F.relu(self.conv3(x))
	x = self.dropout(x)
	x = F.relu(self.conv4(x))
	x = self.dropout(x)
	x = x.view(-1, 14848)
	x = F.relu(self.linear1(x)) 
	x = self.dropout(x)
	x = F.softmax(self.linear2(x), 1)
	return x
</code></pre>

klory: <p>Try not to always use the same <code>dropout</code> layers, using <code>F.dropout()</code>instead</p>

gusido: <p>Ran into the same issue. Did you find an answer?</p>

javidan: <p>I also have same problem. Any idea?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> would be the correct loss function in PyTorch for a multi-class classification/segmentation use case. The original author was using a softmax layer on the output in the model, while <code>nn.CrossEntropyLoss</code> expects raw logits.</p>