Loss computation order changes the results

username: lia_b
cooked: <p>I’m not sure if this is a bug.</p>
<blockquote>
<p>sup_pred = model(supervised_image)<br>
unsup_pred = model(unsupervised_image)</p>
</blockquote>
<p>keeping this constant but in below changing the order:</p>
<blockquote>
<p>loss_unsup = lossfn(unsup_pred, unsup_target)<br>
loss_sup = lossfn(sup_pred, sup_target)</p>
</blockquote>
<p>vs.</p>
<blockquote>
<p>loss_sup = lossfn(sup_pred, sup_target)<br>
loss_unsup = lossfn(unsup_pred, unsup_target)</p>
</blockquote>
<p>if I flip these I get widely different results in accuracy during evaluation.  the</p>
<p>opt.zero_grad()<br>
loss.backward()<br>
opt.step()</p>
<p>remain the same. why does this happen?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is most likely some issue in your code that has side effect.<br>
What is the loss function you’re using? Do you modify any Tensor or state inplace during that computation?<br>
Also what about the rest of the code between the two? And how do you check that you get different results?</p>

username: lia_b
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="96311">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>What is the loss function you’re using? Do you modify any Tensor or state inplace during that computation?</p>
</blockquote>
</aside>
<p>Thank you for your reply.</p>
<p>I am using cross entropy loss in both cases (nn.CrossEntropyLoss()).</p>
<p>I don’t modify any states of the model, I don’t update any tensors either.<br>
My evaluation script is pretty standard. with torch.no_grad(): model.eval() …</p>
<p>For your reference, this is pretty much my code:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/vfdev-5/FixMatch-pytorch/blob/master/main_fixmatch.py#L105">
  <header class="source">

      <a href="https://github.com/vfdev-5/FixMatch-pytorch/blob/master/main_fixmatch.py#L105" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/vfdev-5/FixMatch-pytorch/blob/master/main_fixmatch.py#L105" target="_blank" rel="noopener nofollow ugc">vfdev-5/FixMatch-pytorch/blob/master/main_fixmatch.py#L105</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="95" style="counter-reset: li-counter 94 ;">
          <li>y_pred_cat = model(x_cat)</li>
          <li>y_pred_cat = deinterleave(y_pred_cat, le)</li>
          <li></li>
          <li>idx1 = len(x)</li>
          <li>idx2 = idx1 + len(weak_x)</li>
          <li>y_pred = y_pred_cat[:idx1, ...]</li>
          <li>y_weak_preds = y_pred_cat[idx1:idx2, ...]  # logits_weak</li>
          <li>y_strong_preds = y_pred_cat[idx2:, ...]  # logits_strong</li>
          <li></li>
          <li># supervised learning:</li>
          <li class="selected">sup_loss = sup_criterion(y_pred, y)</li>
          <li></li>
          <li># unsupervised learning:</li>
          <li>y_weak_probas = torch.softmax(y_weak_preds, dim=1).detach()</li>
          <li>y_pseudo = y_weak_probas.argmax(dim=1)</li>
          <li>max_y_weak_probas, _ = y_weak_probas.max(dim=1)</li>
          <li>unsup_loss_mask = (</li>
          <li>    max_y_weak_probas &gt;= engine.state.confidence_threshold</li>
          <li>).float()</li>
          <li>unsup_loss = (</li>
          <li>    unsup_criterion(y_strong_preds, y_pseudo) * unsup_loss_mask</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>there are a lot of functions which are not present in that script (called from elsewhere) but you can assume all losses/criteria are xent</p>
<p>when I flip the sup_loss with unsup_loss, results change.</p>

username: albanD
cooked: <p>Do I understand correctly from your answer " My evaluation script is pretty standard. with torch.no_grad(): model.eval() …" that you compare the results if the two behave exactly the same at evaluation time?</p>
<p>I am afraid floating point numbers are not associative and doing things in a different order will lead to difference in the result.<br>
While you should not see it for one iteration, when you do training, gradient descent based optimization will increase these small differences and you will most likely converge to a different point.<br>
If you model is well behaved, this other point should have similar properties though so that’s not a problem in ML in general.</p>

username: lia_b
cooked: <blockquote>
<p>Do I understand correctly from your answer " My evaluation script is pretty standard. with torch.no_grad(): model.eval() …" that you compare the results if the two behave exactly the same at evaluation time?</p>
</blockquote>
<p>Yes.</p>
<blockquote>
<p>I am afraid floating point numbers are not associative and doing things in a different order will lead to difference in the result.</p>
</blockquote>
<p>Good to know. Very interesting, since these where completely distinct calculations. I think as you say, my training is not that stable, if it leads to this gap.</p>
<p>Thank you for your help!</p>