RMSE loss function

ddd24: <p>Hi all,</p>
<p>I would like to use the RMSE loss instead of MSE. From what I saw in pytorch documentation, there is no build-in function. Any ideas how this could be implemented?</p>

ptrblck: <p>Wouldn’t it work, if you just call <code>torch.sqrt()</code> in <code>nn.MSELoss</code>?</p>
<pre><code class="lang-auto">x = Variable(torch.randn(5, 10), requires_grad=True)
y = Variable(torch.randn(5, 10))

criterion = nn.MSELoss()
loss = torch.sqrt(criterion(x, y))
loss.backward()
print(x.grad)
</code></pre>

pierre: <p>The solution of <a class="mention" href="/u/ptrblck">@ptrblck</a> is the best I think (because the simplest one).<br>
For the fun, you can also do the following ones:</p>
<pre><code class="lang-auto"># create a function (this my favorite choice)
def RMSELoss(yhat,y):
    return torch.sqrt(torch.mean((yhat-y)**2))

criterion = RMSELoss
loss = criterion(yhat,y)
</code></pre>
<pre><code class="lang-auto"># create a nn class (just-for-fun choice :-) 
class RMSELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()
        
    def forward(self,yhat,y):
        return torch.sqrt(self.mse(yhat,y))

criterion = RMSELoss()
loss = criterion(yhat,y)
</code></pre>

YannDubs1: <p>You should be careful with <code>NaN</code> which will appear if the <code>mse=0</code>. Something like this would probably be better :</p>
<pre><code class="lang-auto">class RMSELoss(nn.Module):
    def __init__(self, eps=1e-6):
        super().__init__()
        self.mse = nn.MSELoss()
        self.eps = eps
        
    def forward(self,yhat,y):
        loss = torch.sqrt(self.mse(yhat,y) + self.eps)
        return loss
</code></pre>

Damien_Lancry: <p>sqrt of 0 is 0, not nan</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.sqrt(torch.zeros(1))
tensor([0.])
</code></pre>

YannDubs1: <aside class="quote no-group" data-username="Damien_Lancry" data-post="5" data-topic="16540">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/damien_lancry/48/11393_2.png" class="avatar"> Damien_Lancry:</div>
<blockquote>
<p>torch.sqrt(torch.zeros(1))</p>
</blockquote>
</aside>
<p>Of course, the issue is during the backward pass as you multiply 0 by infinity (derivative of sqrt at 0).</p>
<pre><code class="lang-auto">&gt;&gt;&gt; mse = nn.MSELoss()
&gt;&gt;&gt; yhat = torch.zeros(1, requires_grad=True)
&gt;&gt;&gt; y = torch.zeros(1)
&gt;&gt;&gt; loss = torch.sqrt(mse(yhat,y))
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; yhat.grad
tensor([nan])
</code></pre>
<p>Using the simple module I wrote above</p>
<pre><code class="lang-auto">&gt;&gt;&gt; rmse = RMSELoss()
&gt;&gt;&gt; yhat = torch.zeros(1, requires_grad=True)
&gt;&gt;&gt; y = torch.zeros(1)
&gt;&gt;&gt; loss = rmse(yhat,y)
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; yhat.grad
tensor([0.])
</code></pre>

windson: <p>Hi, I wonder if that’s exactly the same as RMSE when dealing with batch size more than 1 tensor.<br>
i.e. target and prediction are [2,0,256,256] tensor<br>
MSE_0 = MSE(prediction[0,:,:,:], target[0,:,:,:])<br>
MSE_1 = MSE(prediction[1,:,:,:], target[2,:,:,:])</p>
<p>RMSE what we want is:<br>
SQRT( MSE_0) + SQRT( MSE_1)<br>
torch.sqrt(nn.MSELoss(x,y)) will give:<br>
SQRT( MSE_0 + MSE_1)<br>
so:<br>
sqrt(M1+M2)  is not equals to  sqrt(M1) + sqrt(M2)</p>
<p>with reduction is even off, we wanna<br>
Mean[ Mean (sqrt (MSE_0) ) + Mean(sqrt (MSE_1) ) ]<br>
what will get with reduction = ‘mean’  instead, I think is:<br>
sqrt (Mean(MSE_0) + Mean(MSE_1) )<br>
so:<br>
[sqrt(M1) / N + sqrt(M2)/N] /2 is not equals to  sqrt (M1/N + M2/N)</p>
<p>please correct me if my understanding is wrong. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

JamesHowlettLA: <p>Try to add eps, such as eps = 1e-8, according to your precision.,</p>

sudhanshu_rastogi: <p>This implementation is according to the definition of RMSE Error</p>
<pre><code class="lang-auto">class RMSELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.eps=1e-6

    def forward(self,ground_truth,prediction):
        loss=torch.mean(torch.sqrt(torch.sum(torch.square(ground_truth-prediction),axis=-1))) + self.eps
        return loss
</code></pre>

Pytorch-Noob: <p>So in summary one one should implement <strong><a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a></strong>’s implementation by simply taking a square root over the mean square error. However, you need to be aware of Nan resulting in the backward pass.</p>
<p>Hence incredibly important to follow <strong><a href="https://discuss.pytorch.org/u/yanndubs1">YannDubs1</a></strong>’s advice and add a very small non-zero number like 1e-6 or even smaller like 1e-8. This does not effect neuron’s weights as this small number has vanishing nature aka neurons just ignore it.</p>
<p>So the <strong>simplest solution</strong> is to add following code to <strong>ptrblck</strong>’s implementation solution:</p>
<p><code>criterion = nn.MSELoss()</code><br>
<code>loss = torch.sqrt(criterion(x, y) + eps) # added eps to be 1e-6</code></p>
<p>Correct this Noob if wrong!</p>