Does iterative inplace addition on multiple losses have different gradient value than just adding everything at once?

Chrystian: <p>I am a bit confused</p>
<p>Let say I have 4 different losses.</p>
<pre><code class="lang-auto">losses_list = [loss1, loss2, loss3, loss4]
</code></pre>
<p>and as usual I want to add them together. Does the following</p>
<pre><code class="lang-auto">final_loss = 0
for loss in losses_list:
    final_loss += loss
</code></pre>
<p>gives different grad result than doing the following below?</p>
<pre><code class="lang-auto">final_loss = loss1 + loss2 + loss3 + loss4
</code></pre>
<p>My current project perform poorly on the first case, and gives a better results on the latter, or am I just imagining things? and both are equals?</p>

KFrank: <p>Hi Chrystian!</p>
<aside class="quote no-group" data-username="Chrystian" data-post="1" data-topic="153873" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chrystian/48/50459_2.png" class="avatar"> Chrystian:</div>
<blockquote>
<pre><code class="lang-auto">final_loss = 0
for loss in losses_list:
    final_loss += loss
</code></pre>
<p>gives different grad result than doing the following below?</p>
<pre><code class="lang-auto">final_loss = loss1 + loss2 + loss3 + loss4
</code></pre>
</blockquote>
</aside>
<p>It is possible for these two approaches to produces results that differ<br>
by a small floating-point round-off error.  The sequence of additions<br>
could be performed in differing orders that would be mathematically<br>
equivalent, but could differ (by a small round-off error) numerically.</p>
<aside class="quote no-group">
<blockquote>
<p>My current project perform poorly on the first case, and gives a better results on the latter, or am I just imagining things? and both are equals?</p>
</blockquote>
</aside>
<p>Neither approach is better than the other, and they are, in some reasonable<br>
sense, equivalent, even if they produce results that differ.</p>
<p>I would phrase it like this:  By some unlucky happenstance it is possible<br>
that the round-off error in one approach nudges the training into a less<br>
desirable path than in the other.  This is a little unlikely, but possible.</p>
<p>But if you rerun your training multiple times starting with, for example,<br>
different random parameter initializations, or select your training batches<br>
with different random samplings from your training set, the two approaches<br>
should produce statistically equivalent sets of results.</p>
<p>Whether some specific bit of round-off error happens to nudge you down a<br>
better or worse training path will get averaged away over multiple training<br>
runs.</p>
<p>Best.</p>
<p>K. Frank</p>

Chrystian: <p>Thank you for replying. Everythings quite clear now. I also want to think that way but the first results difference quite extreme (weird unlucky moment I guess). It took a whole day to rerun my project and the project also have stochastic augmentation element to it. But after getting 3 results for each method. I couldn’t see any obvious pattern. Both methods have the same performance range. Again, thank you for your time. I really appreciate it.</p>