Using Autocast, returns cannot be converted to type at::Half without overflow: -1e+25

nikkivalenh: <p>I am new in training models, and currently I am trying to train a model using autocast (mixed-precision). I have tried looking at other forums about this error, only 1 solution and it didn’t really help. Any ideas will be accepted. Thank you.</p>
<p>Terminal output :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6bb4b0133db2d1fbadd88f99fd7e51b3376e2fa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a6bb4b0133db2d1fbadd88f99fd7e51b3376e2fa" title="Screenshot 2022-03-15 110814"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6bb4b0133db2d1fbadd88f99fd7e51b3376e2fa_2_690x235.jpeg" alt="Screenshot 2022-03-15 110814" data-base62-sha1="nMYx4ExvuU1wdUOWwulyjVCFMPE" width="690" height="235" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6bb4b0133db2d1fbadd88f99fd7e51b3376e2fa_2_690x235.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6bb4b0133db2d1fbadd88f99fd7e51b3376e2fa_2_1035x352.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6bb4b0133db2d1fbadd88f99fd7e51b3376e2fa_2_1380x470.jpeg 2x" data-dominant-color="3C1831"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-03-15 110814</span><span class="informations">1920×654 136 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Code with error :</p>
<pre><code class="lang-auto">    def forward(self, bert, query, mask):
        bert_embed = bert.unsqueeze(1).expand(-1, query.size(1), -1, -1)
        query_embed = query.unsqueeze(2).expand(-1, -1, bert.size(-2), -1)
        fuse = torch.cat([bert_embed, query_embed], dim=-1)
        x = self.W(fuse)

        x = self.v(torch.tanh(x)).squeeze(-1)
        mask = mask.unsqueeze(1).expand(-1, x.size(1), -1)
        x[~mask]=-1e+25 if attn_logits.dtype == torch.float32 else -1e+4
        x = x.softmax(dim=-1)

        return x
</code></pre>
<p>Used autocast/autograd/mixed precision in this code :</p>
<pre><code class="lang-auto">with autocast():
                # forward step
                entity_logits, entity_bdy = model(encodings=batch['encodings'], context_masks=batch['context_masks'],
                                                token_masks_bool=batch['token_masks_bool'], token_masks=batch['token_masks'], 
                                                pos_encoding = batch['pos_encoding'], wordvec_encoding = batch['wordvec_encoding'], 
                                                char_encoding = batch['char_encoding'], token_masks_char = batch['token_masks_char'], char_count = batch['char_count'])

                # compute loss and optimize parameters
                batch_loss = compute_loss.compute(entity_logits=entity_logits, entity_bdy=entity_bdy, entity_types=batch['gold_entity_types'], entity_spans_token=batch['gold_entity_spans_token'], entity_masks=batch['gold_entity_masks'])
            
                # logging
                iteration += 1
                global_iteration = epoch * updates_epoch + iteration
                epoch_loss += batch_loss / self.args.train_batch_size

                if global_iteration % self.args.train_log_iter == 0:
                    self._log_train(optimizer, batch_loss, epoch, iteration, global_iteration, dataset.label)
</code></pre>

ptrblck: <p>Your check seems to be wrong as:</p>
<pre><code class="lang-python">x[~mask]=-1e+25 if attn_logits.dtype == torch.float32 else -1e+4
</code></pre>
<p>tries to addign <code>-1e-25</code> to a <code>float16</code> tensor, which is not representable.</p>

nikkivalenh: <p>What would be representable in <code>float16</code> tensor? Can you give an example please?</p>
<p>Note : Revision for the code :</p>
<pre><code class="lang-auto">x[~mask]=-1e+25 if attn_logits.dtype == torch.float32 else -1e+4
</code></pre>
<p>is supposed to be :</p>
<pre><code class="lang-auto">x[~mask]=-1e+25 
</code></pre>

ptrblck: <p><code>-65504</code> would be the smallest number es described in <a href="https://en.wikipedia.org/wiki/Half-precision_floating-point_format">Half-precision floating point format</a>:</p>
<pre><code class="lang-python">x[torch.randint(0, 2, (10,))] = -65504 # works
x[torch.randint(0, 2, (10,))] = -65505 # fails
</code></pre>