Autocast error: expected scalar type Half but found Float

username: thauptmann
cooked: <p>Hello all,</p>
<p>I’m trying to add AMP to the source code from a paper I read recently, but it crashes in the forward pass with the error message “RuntimeError: expected scalar type Half but found Float”. And I hope you can help me with this. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The specs I’m using:<br>
Ubuntu 20.4<br>
Nvidia driver: 450.36.06<br>
Cuda: 10.2<br>
cudnn: 7.6.5<br>
PyTorch-Nighlty : 1.7.0.dev20200721</p>
<p>The network is fairly simple:</p>
<pre><code class="lang-auto">scaler = GradScaler()
for epoch in trange(epochs):
    for (dataE, dataM, dataC, target) in trainLoader:
        with autocast():
            zex = AutoencoderE(dataE)
            zmx = AutoencoderM(dataM)
            zcx = AutoencoderC(dataC)
        
            ...
        
        scaler.scale(loss).backward()
        
        for optim in (optim_clas, optimE, optimM, optimC):
            scaler.step(optim)

        scaler.update()
</code></pre>
<p>The network classes:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, input_size, output_size, dropout_rate):
        super(Encoder, self).__init__()
        self.En = torch.nn.Sequential(
            nn.Linear(input_size, output_size),
            nn.BatchNorm1d(output_size),
            nn.ReLU(),
            nn.Dropout(dropout_rate))

    def forward(self, x):
        output = self.En(x)
        return output


class Classifier(nn.Module):
    def __init__(self, input_size, rate):
        super(Classifier, self).__init__()
        self.FC = torch.nn.Sequential(
            nn.Linear(input_size, 1),
            nn.Sigmoid())

    def forward(self, x):
        return self.FC(x)

</code></pre>
<p>Stacktrace:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/media/tony/Volume/work/multi-omics/multi-omics_analysis/molti-omics_analysis_cetuximab.py", line 325, in &lt;module&gt;
    main()
  File "/media/tony/Volume/work/multi-omics/multi-omics_analysis/molti-omics_analysis_cetuximab.py", line 240, in main
    zex = AutoencoderE(dataE)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/tony/Volume/work/multi-omics/multi-omics_analysis/encoder.py", line 14, in forward
    output = self.En(x)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/tony/anaconda3/envs/multi-omics/lib/python3.7/site-packages/torch/nn/functional.py", line 1682, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: expected scalar type Half but found Float

Process finished with exit code 1
</code></pre>

username: Sunshine352
cooked: <p><strong>Data Type Inconsistency</strong><br>
scalar type is not Half (torch.float16), but float (torch.float32)<br>
You should convert scalar to Half like this:<br>
scalar = scalar.to(torch.float16)</p>

username: thauptmann
cooked: <p>Thanks. I changed it, but now I get “RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED”</p>

username: Sunshine352
cooked: <aside class="quote no-group" data-username="thauptmann" data-post="3" data-topic="90659">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thauptmann/48/53206_2.png" class="avatar"> thauptmann:</div>
<blockquote>
<p>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED</p>
</blockquote>
</aside>
<p>This Error can be caused by various factors, which may be solved according to solutions returned by Search engines like Google or Baidu.</p>

username: thauptmann
cooked: <p>I found my error. I was thinking that tensor.to(device) works inplace. Now it works without an explicit conversion</p>