Self.scaler.step(self.d_optimizer): AssertionError: No inf checks were recorded for this optimizer

v-moayman: <p>I am new to pytorch and am trying to reduce the consumption usage of GPUs. What I am trying to do is to update the weights manually. In this sense, I am getting the new gradient values. Then, I update the weights as follows:</p>
<pre><code class="lang-auto"> grads = torch.autograd.grad(
                        d_loss, weights.values(), create_graph=True, allow_unused=True
                    )
weights = OrderedDict(
                        (name, param - grad) if grad is not None else (name, param)
                        for ((name, param), grad) in zip(weights.items(), grads)
                    )
</code></pre>
<p>The problem here is that I do need to update the weights without making the gradient to be true, as it increases the GPU consumption usage without a need for it.</p>
<p>When I changed the second line as follows:</p>
<pre><code class="lang-auto">with torch.no_grad():
                            for ((name, param), grad) in zip(weights.items(), grads):
                                if grad is not None:
                                    param -= grad 
</code></pre>
<p>However, it gives me the following message:</p>
<pre><code class="lang-auto">assert len(optimizer_state["found_inf_per_device"]) &gt; 0, "No inf checks were recorded for this optimizer."
AssertionError: No inf checks were recorded for this optimizer
</code></pre>
<p>What I do not understand why this even occurs. I just tried to make the updates inplace without a need for the gradient to be true. Any help?</p>

ptrblck: <aside class="quote no-group" data-username="v-moayman" data-post="1" data-topic="158800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/e47774/48.png" class="avatar"> v-moayman:</div>
<blockquote>
<p>The problem here is that I do need to update the weights without making the gradient to be true, as it increases the GPU consumption usage without a need for it.</p>
</blockquote>
</aside>
<p>I don’t understand this claim.<br>
There shouldn’t be any difference in GPU memory usage if you calculate the gradients explicitly via <code>torch.autograd.grad</code> or if you allow the <code>backward()</code> call to store these gradients in the <code>.grad</code> attributes of the trainable parameters. In the end you need to store the gradients somewhere, so could you explain which operation uses GPU memory without the need for it?</p>
<aside class="quote no-group" data-username="v-moayman" data-post="1" data-topic="158800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/e47774/48.png" class="avatar"> v-moayman:</div>
<blockquote>
<p>What I do not understand why this even occurs.</p>
</blockquote>
</aside>
<p>You are most likely trying to use <code>torch.cuda.amp</code> and call <code>scaler.step(optimizer)</code> somewhere which internally will call <code>optimizer.step()</code> and fail as it’s not needed.<br>
Since you are already using a manual approach of directly updating the parameters (instead of letting the optimizer do it) you might also want to consider using a manual loss scaling approach.</p>

v-moayman: <p>Thank you for your reply,<br>
I mean that the line that causes of updating the weights leads to a huge increase in the GPU consumption. Now, since the updates do not need the grad to be true, I made it under with torch. no_grad()<br>
Regarding the updating the parameters, this the the snippet of my code:</p>
<pre><code class="lang-auto">
for outer_step in range(self.outer_steps):
                for inner_step in range(self.n_inner_steps):
                    # forward
                    d_losses, _, bert_outputs = model.forward_with_params(
                       .....
                    )
                    d_loss = d_losses.mean()
                  
                    # backward
                    grads = torch.autograd.grad(
                        d_loss, weights.values(), create_graph=True, allow_unused=True
                    )
    
                    # # update parameters (SGD)
                    with torch.no_grad():
                        for ((name, param), grad) in zip(weights.items(), grads):
                            if grad is not None:
                               param -= grad
                
                    del grads 
                    del d_loss
                    torch.cuda.empty_cache()

                    if self.accum_loss or (inner_step + 1) == self.n_inner_steps:
                        # loss of updated parameters (after each gradient step)
                        with amp.autocast(dtype=self.dtype, enabled=self.use_amp):
                            after_losses, after_logits, _ = model.forward_with_params(
                                input_ids=input_ids.to(self.device),
                                attention_mask=attention_mask.to(self.device),
                                labels=labels.to(self.device),
                                weights=weights,
                            )
                            after_loss = after_losses.mean()
                            loss += after_loss

                cur_after_loss += after_loss.item() * batch_size
                cur_after_correct += (
                    after_logits.cpu().argmax(1).eq(labels).sum().item()
                )

                
                self.d_optimizer.zero_grad()
                # backward
                self.scaler.scale(loss).backward()
                # unscale gradients (for gradient clipping)
                self.scaler.unscale_(self.d_optimizer)
                # gradient cliping
                torch.nn.utils.clip_grad_norm_(
                    self.optimize_param_list, self.max_grad_norm
                )
                self.scaler.step(self.d_optimizer)
                self.scaler.update()
                self.d_scheduler.step()
</code></pre>
<p>What is interesting here is that when I replace the snippet code of updating the weights<br>
as shown in the post, it works.</p>

ptrblck: <aside class="quote no-group" data-username="v-moayman" data-post="3" data-topic="158800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/e47774/48.png" class="avatar"> v-moayman:</div>
<blockquote>
<p>I mean that the line that causes of updating the weights leads to a huge increase in the GPU consumption.</p>
</blockquote>
</aside>
<p>Which optimizer are you using? If the optimizer itself uses running stats (e.g. such as <code>Adam</code>) an increase in memory would be expected since the optimizer needs to allocate these stats and track them. In your manual update you would then skip these running stats and would most likely fall back to a plain <code>SGD</code> update rule.</p>

v-moayman: <p>It is apparent that Adam is memory hungry, but would it reserve more than 20 GB from GPU memory?</p>

ptrblck: <p>It would depend on the parameter count but 20GB sounds for the majority of models too large, so could you share a minimal, executable code snippet which would show these 20GB increase?</p>

v-moayman: <p>The GPU increase was due to Adam usage for the GPU. I guess you were right. Thank you!</p>