Why do we use RMSE instead of MSE?

serdarrader: <p>Why do we calculate square root of MSE since minimizing MSE is the same as minimizing RMSE ? Is it because of numerical stability or something ? Or to avoid exploding gradient which can result from bigger loss function values?</p>

KFrank: <p>Hello Serdar!</p>
<aside class="quote no-group" data-username="serdarrader" data-post="1" data-topic="99010" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e274bd/48.png" class="avatar"> serdarrader:</div>
<blockquote>
<p>Why do we calculate square root of MSE since minimizing MSE is the same as minimizing RMSE ?</p>
</blockquote>
</aside>
<p>First a comment:  I would lean towards using the mean-squared-error<br>
(MSE), as it is a “more natural” measure of error (whatever that might<br>
mean).</p>
<p>(Just to be clear “RMSE” is an acronym for “root-mean-squared-error”,<br>
and is equal to <code>sqrt (MSE)</code>.)</p>
<p>Now for some concrete technical differences:</p>
<p>Consider a single variable, <code>x</code>, and minimizing <code>x**2</code> with respect to<br>
<code>x</code> using gradient descent.  Note that <code>sqrt (x**2) = abs (x)</code>.  <code>x**2</code><br>
is the one-dimensional version of MSE, and <code>abs (x) = sqrt (x**2)</code><br>
is the one-dimensional version of RMSE.</p>
<p>Both <code>x**2</code> and <code>abs (x)</code> are minimized when <code>x = 0</code> (at which point<br>
both equal zero).  The gradient of <code>x**2</code> is “softer” in that is gets<br>
smaller (and approaches zero) as <code>x</code> gets closer to 0.  In contrast,<br>
the gradient of <code>abs (x)</code> is always either <code>+1</code> or <code>-1</code>, and doesn’t<br>
change in magnitude as <code>x</code> approaches zero.</p>
<p>When <code>x</code> is large (greater than 1/2), <code>x**2</code> will have the larger gradient,<br>
and, using gradient descent, drive you towards the minimum at zero<br>
more rapidly.  But when <code>x</code> is small, <code>abs (x)</code> will have the larger<br>
gradient.</p>
<p>Unless you expect <code>x</code> to start out very large, you might expect<br>
minimization of <code>abs (x)</code> to proceed more rapidly because its<br>
gradient doesn’t get smaller.  On the other hand, because the<br>
magnitude of its gradient stays the same, once near the <code>x = 0</code><br>
minimum, you might expect gradient descent to jump back and<br>
forth from positive <code>x</code> to negative <code>x</code> back to positive <code>x</code>, and so on,<br>
without making further progress towards <code>x = 0</code>.</p>
<p>So …   Pick your poison.</p>
<p>(All of these effects can be addressed to some degree by using<br>
variants of plain-vanilla gradient descent, such as adding momentum,<br>
using an optimizer such as <code>Adam</code>, and/or using a learning-rate<br>
scheduler.)</p>
<p>Of course, the realistic case of using either MSE or RMSE as the<br>
loss function to be applied to the output of a complicated network<br>
is much more involved, but, at some level, the above comments still<br>
apply.</p>
<p>Best.</p>
<p>K. Frank</p>

Pengzhangzhi: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="99010">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>be addressed to some degree by using<br>
variants of plain-vanilla gradient descent, such as adding momentum,<br>
using an optimizer such as <code>Adam</code> , and/or using a learning-rate<br>
scheduler.)</p>
<p>Of course, the realistic case</p>
</blockquote>
</aside>
<p>That is a very outstanding answer!</p>

elonjigar: <ul>
<li>The RMSE is an indication of the noise levels in the scale of standard deviations.</li>
<li>The <a href="http://net-informations.com/ds/psa/rmse.htm" rel="noopener nofollow ugc">RMSE</a> has nice mathematical properties for fast calculations (Its gradient is linear and propagates easily).</li>
</ul>