Custom Loss Function(derivative not implemented

username: Abdulrahman
cooked: <p>When i tried to reimplement MSELoss proposed here <a href="https://discuss.pytorch.org/t/custom-loss-functions/29387">Loss Pytroch Implementation</a> with a real example, it gives <strong>RuntimeError: derivative for argmax is not implemented</strong> although i take argmax with nn.MSELoss and worked.</p>
<p>Code  def my_MSELoss(predict, true):<br>
return ((predict - true)**2).mean()</p>
<p>for epoch in range(5):<br>
losses = 0.0<br>
for data, label in train_loader:<br>
network.zero_grad()<br>
predict = network(data.unsqueeze(1).float())<br>
print(predict.shape)<br>
<span class="hashtag">#predict</span> = predict.argmax(axis = 1)<br>
loss = my_MSELoss(predict.argmax(axis = 1), label.float())<br>
print(loss)<br>
loss.backward()<br>
optimizer.step()<br>
loss = loss.item()<br>
losses += loss<br>
print(“For epoch: {}, losses are: {}”.format(epoch + 1, losses))</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using? If I remember correctly, this bug was recently fixed by <a class="mention" href="/u/alband">@albanD</a> in <a href="https://github.com/pytorch/pytorch/pull/37789">this PR</a> and I get an error using your <code>my_MSELoss</code> function as:</p>
<blockquote>
<p>RuntimeError: derivative for argmax is not implemented</p>
</blockquote>

username: Abdulrahman
cooked: <p>I’m Sorry, but i don’t know how to A dd <code>argmin</code> , <code>argmax</code> and <code>argsort</code> to the list of non-differentiable functions</p>
<p>For torch version it’s 1.5 running on Google Colab<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4ff4105487c49323d56902a1d6d28460209e4f1e.png" alt="Version" data-base62-sha1="bpiCIhCBEwIDmQ1M6zBSqJBjwJw" width="407" height="98"></p>

username: ptrblck
cooked: <p>The idea is that none of your methods should work, as the derivative of <code>argmax</code> is not implemented.<br>
It seems to be a bug that one of your approaches seems to have worked.</p>

username: Abdulrahman
cooked: <p>When i tried using max() it throw another error</p>
<pre><code class="lang-auto">Code is:
def my_MSELoss(predict, true):
    return ((predict - true)**2).mean()
    
for epoch in range(5):
    losses = 0.0
    for data, label in train_loader:
        #.float() to convert data into weights type
        #you can know type of weights through: 
        #next(model.conv1.parameters()).dtype
        network.zero_grad()
        predict = network(data.unsqueeze(1).float())
        print(predict.shape)
        #predict = predict.argmax(axis = 1)
        _, y_preds = predict.max(axis = 1)
        print(y_preds.shape)
        loss = my_MSELoss(y_preds, label.float())
        print(loss)
        loss.backward()
        optimizer.step()
        loss = loss.item()
        losses += loss
    print("For epoch: {}, losses are: {}".format(epoch + 1, losses))

Output is:
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

username: ptrblck
cooked: <p><code>y_preds</code> is still the <code>argmax</code>. You could get the gradients for the first output (which is the max value).<br>
In that case <code>max</code> would be differentiable and the gradients would just flow back to the maximum value while all other values will get a 0 gradient.<br>
<code>Argmax</code> on the other hand is not differentiable as it return integer values. So you can’t get gradients for it.</p>