Training loss issue

username: kartheek
cooked: <p>when i m training resnet modelin pytorch i got the traing loss is constant in every time …<br>
pytorch:<br>
elif self.model_type == ‘resnet’:</p>
<pre><code>            self.model = models.resnet50()

            self.model.adaptiveavgpool = nn.AdaptiveAvgPool2d(self.model.fc.out_features)

            self.model.fc = nn.Linear(in_features=self.model.fc.in_features, out_features=self.output_classes)
             self.model.activation = nn.Softmax(self.model.fc)
</code></pre>
<p>keras:<br>
elif self.model_type in [‘resnet50’]:<br>
# self.model.layers.pop()<br>
model_output = self.model.output<br>
model_output = GlobalAveragePooling2D()(model_output)<br>
predictions = Dense(self.output_classes, activation=‘softmax’)(model_output)<br>
self.model = Model(self.model.input, predictions)</p>
<p>we are doing changing keras to pytorch…so we have an issue on that…</p>
<p>can you please give your suggestions in this…tpoic???</p>
<p>i got this like<br>
Epoch: [0] Train Loss: [1.636952]  Valid Loss: [1.576530]<br>
Epoch: [1] Train Loss: [1.636952]  Valid Loss: [1.404914]<br>
Epoch: [2] Train Loss: [1.636952]  Valid Loss: [1.395322]<br>
Epoch: [3] Train Loss: [1.636952]  Valid Loss: [1.391647]<br>
Epoch: [4] Train Loss: [1.636952]  Valid Loss: [1.390293]<br>
Epoch: [5] Train Loss: [1.636952]  Valid Loss: [1.393284]<br>
Epoch: [6] Train Loss: [1.636952]  Valid Loss: [1.395637]<br>
Epoch: [7] Train Loss: [1.636952]  Valid Loss: [1.396015]</p>
<p>train the model:</p>
<p>for epoch in range(num_epochs):</p>
<pre><code>                    # set the model in train mode

                    self.model.train() 

                    self.model.optimizer.zero_grad()                     

                    # training_input_data = training_input_data.resize_([training_batch_size, 3, 224, 224])

                    training_ground_truth = training_ground_truth.resize_([training_batch_size])

                    self.model = self.model.double()

                    outputs = self.model(training_input_data.double())      

                    sm=torch.nn.Softmax()

                    probabilities=sm(outputs)

                    print("predicted probabilities while testing",probabilities) 

                    loss = self.model.criterion(outputs, training_ground_truth.long())

                    loss.backward()

                    self.model.optimizer.step()

                    train_losses.append(loss.item()</code></pre>

username: ptrblck
cooked: <p>These seem to be a few issues in your PyTorch code:</p>
<ul>
<li>you are assigning layers to new (and thus) unused attributes. <code>model.adaptiveavgpool</code> does not exist in <code>resnet50</code>, so you are creating a new attribute (maybe you want to change <code>model.avgpool</code>?).</li>
<li>same for <code>model.activation</code>
</li>
<li>If your criterion is <code>nn.CrossEntropyLoss</code>, pass the raw logits without the softmax to the criterion.<br>
Internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be applied.</li>
</ul>

username: kartheek
cooked: <p>that issue is when i am running a model on CPU it will changes training loss and accuracy changes but when i go to GPU it will constant  .??<br>
can you please give the solution for this??<br>
as soon as possible</p>

username: ptrblck
cooked: <p>Could you post your updated model and training code so that we could have another look?</p>
<p>You can post source code directly by wrapping it into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: kartheek
cooked: <p>there will be an error on while I m putting model.avgpool this?<br>
i.e out of memory<br>
self.model = models.resnet50()</p>
<pre><code>        self.model.avgpool = nn.AdaptiveAvgPool2d(self.model.fc.out_features)

        self.model.fc = nn.Linear(in_features=self.model.fc.in_features, out_features=self.output_classes)
</code></pre>
<h1>train the model</h1>
<pre><code>                for epoch in range(num_epochs):

                    # set the model in train mode

                    self.model.train() 

                    self.model.optimizer.zero_grad()                     

                    # training_input_data = training_input_data.resize_([training_batch_size, 3, 224, 224])

                    training_ground_truth = training_ground_truth.resize_([training_batch_size])

                    self.model = self.model.double()

                    outputs = self.model(training_input_data.double())      

                    sm=torch.nn.Softmax()

                    probabilities=sm(outputs)

                    print("predicted probabilities while testing",probabilities) 

                    loss = self.model.criterion(outputs, training_ground_truth.long())

                    loss.backward()

                    self.model.optimizer.step()

                    train_losses.append(loss.item())</code></pre>

username: kartheek
cooked: <p>after training on vgg16 architecture  the losses are different but when i run on Resnet50 architecture  its always constant …<br>
can you please give the solution for that ???</p>

username: ptrblck
cooked: <p>The provided code shouldn’t work, so please make sure you are really executing the posted code.</p>
<p>This line of code:</p>
<pre><code class="lang-python">self.model.avgpool = nn.AdaptiveAvgPool2d(self.model.fc.out_features)
</code></pre>
<p>will initialize an adaptive pooling layer with a spatial output size of 1000, which won’t work using the following <code>self.model.fc</code> layer with <code>in_features = 2048</code>.</p>