Discrepancy in RMSE Calculation Results between Functions

ababey: <p>Hello all,</p>
<p>I’m encountering an issue in my PyTorch code involving RMSE calculations. I have two functions that calculate the RMSE between corresponding feature maps in two lists. Both functions should do the same ; compute the RMSE for a list of 5 tensors. The difference is that in the second one I perform manually the iteration over the batch size.</p>
<p>The strange part is that while both functions seem logically equivalent and work correctly, they yield significantly different RMSE values.</p>
<p>Here’s a brief overview of my code and the functions with print of the loss they compute:</p>
<pre><code class="lang-auto">def calculate_rmse(A, B):
    diff_squared = (A - B)**2
    mean_squared_error = torch.mean(diff_squared)
    rmse = torch.sqrt(mean_squared_error)
    return rmse



def calculate_total_rmse_list(original_feature_maps_list, reconstructed_feature_maps_list):
    total_rmse = 0.0
    for i in range(len(original_feature_maps_list)):
        rmse = calculate_rmse(original_feature_maps_list[i], reconstructed_feature_maps_list[I])
        total_rmse += rmse
    return total_rmse
Epoch 1/200, Loss: 6.661827330362229

def calculate_total_rmse_list2(original_feature_maps_list, reconstructed_feature_maps_list, batch_size):
    total_rmse = 0.0
    for i in range(len(original_feature_maps_list)):
        original_feature_map = original_feature_maps_list[I]
        reconstructed_feature_maps = reconstructed_feature_maps_list[I]
        rmse_batch = 0.0
        for j in range(batch_size):
            rmse = calculate_rmse(original_feature_map[j,:,:,:], reconstructed_feature_maps[j,:,:,:])
            rmse_batch += rmse
        total_rmse += rmse_batch
    return total_rmse

Epoch 1/200, Loss: 418.642820085798
</code></pre>
<p>I do not understand why there is a difference between the two function, same data, same code. I try the second function to debug the first one, but it gave me more issue to understand. It seems that pytorch do not handle a 4 dim tensors difference but I know it can.</p>
<p>Can someone help to understand why there is a difference?<code>Preformatted text</code></p>

KFrank: <p>Hi Ababey!</p>
<aside class="quote no-group" data-username="ababey" data-post="1" data-topic="187464" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ababey/48/59680_2.png" class="avatar"> ababey:</div>
<blockquote>
<pre><code class="lang-auto">def calculate_total_rmse_list(original_feature_maps_list, reconstructed_feature_maps_list):
    total_rmse = 0.0
    for i in range(len(original_feature_maps_list)):
        rmse = calculate_rmse(original_feature_maps_list[i], reconstructed_feature_maps_list[I])
        total_rmse += rmse
    return total_rmse
Epoch 1/200, Loss: 6.661827330362229

def calculate_total_rmse_list2(original_feature_maps_list, reconstructed_feature_maps_list, batch_size):
    total_rmse = 0.0
    for i in range(len(original_feature_maps_list)):
        original_feature_map = original_feature_maps_list[I]
        reconstructed_feature_maps = reconstructed_feature_maps_list[I]
        rmse_batch = 0.0
        for j in range(batch_size):
            rmse = calculate_rmse(original_feature_map[j,:,:,:], reconstructed_feature_maps[j,:,:,:])
            rmse_batch += rmse
        total_rmse += rmse_batch
    return total_rmse

Epoch 1/200, Loss: 418.642820085798
</code></pre>
</blockquote>
</aside>
<p>You have three things going on here:</p>
<p>First uppercase’<code>I</code>’ isn’t defined anywhere in the code you posted.  You<br>
might have a typo and meant lowercase ‘<code>i</code>’.  In any event, an inconsistent<br>
value for <code>I</code> would definitely mess things up.</p>
<p>Second, when you compute <code>calculate_rmse()</code> in your first version for<br>
<code>original_feature_maps_list[i]</code>, <code>torch.mean()</code> divides by the number<br>
of elements of <code>original_feature_maps_list[i]</code>.  In the second version,<br>
<code>torch.mean()</code> only divides by the smaller number of elements in<br>
<code>original_feature_map[j,:,:,:]</code>.</p>
<p>Finally, your second version sums up <code>rmse</code> to form <code>rmse_batch</code> <em>after</em> the<br>
square root has been taken, while in the first version the squared errors of<br>
the batch elements are summed up <em>before</em> taking the square root, which is<br>
not equivalent.</p>
<p>If you’re still having issues after sorting through the above, please post a<br>
fully-self-contained (using hard-coded or random data), runnable script that<br>
reproduces your issue, together the results you get when you run that script.</p>
<p>Good luck!</p>
<p>K. Frank</p>