How to skip update if some of mini-batch is not fit on condition?

tolry418: <p>Hello.<br>
I want to skip update and go to next iteration. For example</p>
<pre><code class="lang-auto">for it, input in dataloader:
    optimizer.zero_grad()
    output = model(input)
    loss = myloss(output, label) 
    if loss.item()==0 or not torch.isfinite(loss):
        loss.backward()
        continue

    loss.backward()
    optimizer.step()

</code></pre>
<p>I want to do above like that.</p>
<p>First , once I got loss as ‘nan’ , and continue to get ‘nan’ from then on.<br>
I think that weight got updated based on gradient by ‘nan’ and continue.<br>
How to solve it?</p>
<p>Second. I make loss 0 on purpose for no updating weight. (To make gradient 0 , weight not gonna change)<br>
Because some of mini-batch is not fit on condition.<br>
For example.<br>
I got 8 mini-batch.</p>
<pre><code class="lang-auto">class myloss():
  def __init__():
        blah, blah...

  def forward(self, x, A_list):
    A_list =  [None, 1, None , 1, 1, 1, 1, 1 ] # for example
    batch_loss = torch.tensor(0.0)
    for a in A_list:
      if not a == None:
         loss = criterion(x)
         batch_loss += loss
      else:
         loss = torch.log(x/x) #( make  loss 0 on purpose.. is it right way?)
         batch_loss +=loss
    return batch_loss
</code></pre>
<p>I don’t know that making loss 0 on purpose is right.<br>
Is someone who have good idea for that?</p>
<p>Thanks in advance.</p>

ptrblck: <aside class="quote no-group" data-username="tolry418" data-post="1" data-topic="156947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tolry418/48/51168_2.png" class="avatar"> tolry418:</div>
<blockquote>
<p>I think that weight got updated based on gradient by ‘nan’ and continue.<br>
How to solve it?</p>
</blockquote>
</aside>
<p>Calling <code>loss.backward()</code> if the loss it not finite and thus contains invalid values wouldn’t make sense, but since <code>optimizer.step()</code> wasn’t called the parameters should not be invalid.<br>
You could verify it by checking all parameters of the model once the update step is skipped.<br>
However, depending where the invalid activations were created in the model, e.g. the running stats of batchnorm layers might have been updated with these NaN/Inf values, which would then break your model in the validation run, so you should try to avoid creating invalid activations at all.</p>
<aside class="quote no-group" data-username="tolry418" data-post="1" data-topic="156947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tolry418/48/51168_2.png" class="avatar"> tolry418:</div>
<blockquote>
<p>Second. I make loss 0 on purpose for no updating weight. (To make gradient 0 , weight not gonna change)</p>
</blockquote>
</aside>
<p>I would not recommend this approach as even if the gradients are zero, the optimizer could still update the parameters if it holds internal states from the last updates (such as <code>Adam</code>).<br>
If you want to skip a batch, skip it explicitly instead of manipulating the gradients.</p>

tolry418: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks your reply.<br>
Summary.  1. Verifying all parameters after model update step is skipped.<br>
2. Optimizer could update with internal states from the last updates…so making gradient 0 is not appropriate.<br>
SO…</p>
<blockquote>
<p>If you want to skip a batch, skip it explicitly instead of manipulating the gradients.</p>
</blockquote>
<p>I am wondering how to skip it explicitly??? Could you give me a example to understand in detail?<br>
Thank you!</p>

ptrblck: <p>Maybe just use a condition on the <code>backward</code> and <code>step</code> call:</p>
<pre><code class="lang-python">if loss.item()!=0 or torch.isfinite(loss):
    loss.backward()
    optimizer.step()
</code></pre>

tolry418: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Really thanks your reply.<br>
But, I am really confusing… I give you another example.<br>
What if A_list =  [None, None, None , None, None, None, None, None ]  instead of [None, 1, None , 1, 1, 1, 1, 1 ]. (A_list is supplementary mean to filter data which is not fit my idea)<br>
According to your answer, if I erase part of  “loss 0 on purpose” , myloss would return nothing like below.<br>
I think it couldn’t do back-propagation because loss is nothing.<br>
On this issue, is it possible to skip update explicitly?<br>
What did i misunderstand?</p>
<pre><code class="lang-auto">class myloss():
  def __init__():
        blah, blah...

  def forward(self, x, A_list):
    A_list =  [None, None, None , None, None, None, None, None ]# for example
    batch_loss = torch.tensor(0.0)
    for a in A_list:
      if not a == None:
         loss = criterion(x)
         batch_loss += loss
    return batch_loss
</code></pre>
<p>And How to initialize batch_loss? torch.tensor(0.0) may be wrong.</p>

tolry418: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I’m so sorry bothering you…Once again, Could you tell me any solution for this?<br>
Thank you.</p>

thecho7: <p>I didn’t understand exactly.<br>
If you need a small chat, message to me</p>

ptrblck: <p>Yes, it should be possible to skip the <code>backward</code> and <code>optimizer.step()</code> calls by e.g. checking if the returned <code>batch_loss</code> tensor has a valid <code>.grad_fn</code> attribute or if it’s set to <code>None</code>.</p>

tolry418: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Really appreciate you. Thecho7 let me know set None if to skip update. Thanks you both of you.</p>