RuntimeError: expected scalar type Float but found Half

shrutee: <p>I am using <code>torch.cuda.amp</code> for mixed precision.</p>
<p>My forward pass calls many functions with their own <code>forward</code> passes.<br>
I tried to decorate all the forward passes in the subsequent functions with <code>torch.cuda.amp.autocast(enabled=True)</code> but the error persists.</p>
<p>Forward pass:</p>
<pre><code class="lang-auto">with torch.cuda.amp.autocast(enabled=True):
                h, chunk, preds, labels = model.forward(batch, alphaSG, device)
                label = labels
                for worker in model.classification_workers:
                    loss = worker.loss_weight * worker.loss(preds[worker.name], label[worker.name])
                    losses[worker.name] = loss
                    tot_loss += loss

                for worker in model.regression_workers:
                    loss = worker.loss_weight * worker.loss(preds[worker.name], label[worker.name])
                    losses[worker.name] = loss
                    tot_loss += loss
</code></pre>
<p>Error I am getting:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-10-f4ce5bf32b0d&gt; in &lt;module&gt;()
   2327 
   2328             with torch.cuda.amp.autocast(enabled=True):
-&gt; 2329                 h, chunk, preds, labels = model.forward(batch, alphaSG, device)
   2330                 label = labels
   2331                 for worker in model.classification_workers:

10 frames
/usr/local/lib/python3.6/dist-packages/torch/cuda/amp/autocast_mode.py in decorate_autocast(*args, **kwargs)
    133         def decorate_autocast(*args, **kwargs):
    134             with self:
--&gt; 135                 return func(*args, **kwargs)
    136         return decorate_autocast
    137 

&lt;ipython-input-10-f4ce5bf32b0d&gt; in forward(self, x, alpha, device)
   1945             # remove key if it exists
   1946             x_.pop('cchunk', None)
-&gt; 1947         h = self.frontend(x_, device)
   1948         if len(h) &gt; 1:
   1949             assert len(h) == 2, len(h)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    556             result = self._slow_forward(*input, **kwargs)
    557         else:
--&gt; 558             result = self.forward(*input, **kwargs)
    559         for hook in self._forward_hooks.values():
    560             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/cuda/amp/autocast_mode.py in decorate_autocast(*args, **kwargs)
    133         def decorate_autocast(*args, **kwargs):
    134             with self:
--&gt; 135                 return func(*args, **kwargs)
    136         return decorate_autocast
    137 

&lt;ipython-input-10-f4ce5bf32b0d&gt; in forward(self, batch, device, mode)
   1827             dskips = []
   1828         for n, block in enumerate(self.blocks):
-&gt; 1829             h = block(h)
   1830             if denseskips and (n + 1) &lt; len(self.blocks):
   1831                 # denseskips happen til the last but one layer

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    556             result = self._slow_forward(*input, **kwargs)
    557         else:
--&gt; 558             result = self.forward(*input, **kwargs)
    559         for hook in self._forward_hooks.values():
    560             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/cuda/amp/autocast_mode.py in decorate_autocast(*args, **kwargs)
    133         def decorate_autocast(*args, **kwargs):
    134             with self:
--&gt; 135                 return func(*args, **kwargs)
    136         return decorate_autocast
    137 

&lt;ipython-input-10-f4ce5bf32b0d&gt; in forward(self, x)
   1494                 P = (pad, pad)
   1495             x = F.pad(x, P, mode=self.pad_mode)
-&gt; 1496         h = self.conv(x)
   1497         if hasattr(self, 'norm'):
   1498             h = forward_norm(h, self.norm)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    556             result = self._slow_forward(*input, **kwargs)
    557         else:
--&gt; 558             result = self.forward(*input, **kwargs)
    559         for hook in self._forward_hooks.values():
    560             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/cuda/amp/autocast_mode.py in decorate_autocast(*args, **kwargs)
    133         def decorate_autocast(*args, **kwargs):
    134             with self:
--&gt; 135                 return func(*args, **kwargs)
    136         return decorate_autocast
    137 

&lt;ipython-input-10-f4ce5bf32b0d&gt; in forward(self, waveforms)
   1334         band=(high-low)[:,0]
   1335 
-&gt; 1336         f_times_t_low = torch.matmul(low, self.n_)
   1337         f_times_t_high = torch.matmul(high, self.n_)
</code></pre>

ptrblck: <p>Could you post a code snippet to reproduce this issue, please?</p>

shrutee: <p>Hi,<br>
Here is the <a href="https://colab.research.google.com/gist/MittalShruti/c4d9f7292b10f2eb6c6130f69529bfd6/mixed-precision-torch-cuda-amp.ipynb?authuser=3" rel="nofollow noopener">code</a></p>
<p>this is 115 lines of code. Please let me know if this wouldn’t help.</p>

ptrblck: <p>The error points towards a device mismatch:</p>
<pre><code class="lang-python">RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_mm
</code></pre>
<p><code>amp</code> needs a GPU to run properly, so you would need to call <code>.to('cuda')</code> on the model and input.<br>
Also, you might need to register <code>self.n_</code> as a buffer via:</p>
<pre><code class="lang-python">self.register_buffer('n_', ...)
</code></pre>
<p>so that it will be also pushed to the device.</p>
<p>Your code doesn’t provide the definition of <code>format_frontend_output</code> so that I cannot verify, if it’s working after these fixes.</p>

shrutee: <p>Thank You!! this solves the error.</p>
<p>Two points -</p>
<ol>
<li>
<p>Where/how did you get the device mismatch error?</p>
</li>
<li>
<p><code>self.n_</code> is already on device, so why use register_buffer again? There are many other variables in the actual code, how do you decide if a parameter should be registered as a buffer or not? Here, in this code, e.g. I have <code>self.window_</code> along with <code>self.n_</code>; why not register it as buffer as well?</p>
</li>
</ol>

ptrblck: <p>I’m glad it solved the error! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<ol>
<li>
<p>The error was raised in my environment. If you have executed the script in Colab, the error message might have been lost?</p>
</li>
<li>
<p>Usually, every tensor, which should be pushed to the same device as the model parameters, but doesn’t require gradients, should be registered as a buffer. On the other hand, if the tensor requires gradients, use <code>nn.Parameter</code>.</p>
</li>
</ol>

Kushagra_Bhatia: <p>Can you please help me out with this I am facing a similar problem. Getting the error:<br>
<code>Expected object of scalar type Float but got scalar type Half for argument #2 'mat2' in call to _th_mm</code></p>
<p>Link to my problem: <a href="https://discuss.pytorch.org/t/expected-object-of-device-type-cuda-but-got-device-type-cpu-for-argument-1-self-in-call-to-th-mm/86921/13" class="inline-onebox">Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_mm</a></p>
<p>Thanks!</p>