Pytorch model FP32 to FP16 using half()- LSTM block is not casted

username: may1
cooked: <p>I am trying to convert some pre-trained models to half precision for deployment.</p>
<p>I tried</p>
<pre><code class="lang-auto">model = model.half()
</code></pre>
<p>and it could cast the basic FP32 CNN model into FP32 well<br>
but not LSTM model</p>
<p>I got this error when FP16 input was passed to torch.nn.LSTM layer after I called model.half()</p>
<pre><code class="lang-auto">  File "/~env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/~env/lib/python3.9/site-packages/torch/nn/modules/rnn.py", line 769, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
RuntimeError: Input and hidden tensors are not the same dtype, found input tensor with Half and hidden tensor with Float
</code></pre>
<p>When I read the docs,<br>
half function can cast all floating point parameters and buffers to half datatype. Right? This method modifies the module in-place.</p>
<p>But why LSTM layer cannot be casted? and How can I cast it to half datatype?</p>

username: ptrblck
cooked: <p>It works for me:</p>
<pre><code class="lang-python"># float32
model = nn.LSTM(10, 20, 2).cuda()
input = torch.randn(5, 3, 10).cuda()
h0 = torch.randn(2, 3, 20).cuda()
c0 = torch.randn(2, 3, 20).cuda()

out, (h, c) = model(input, (h0, c0))
print(out.dtype, h.dtype, c.dtype)
# torch.float32 torch.float32 torch.float32

# float16
model.half()
input = input.half()
h0 = h0.half()
c0 = c0.half()

out, (h, c) = model(input, (h0, c0))
print(out.dtype, h.dtype, c.dtype)
# torch.float16 torch.float16 torch.float16
</code></pre>
<p>and based on the error message I would assume you are not casting all inputs to <code>float16</code>.</p>

username: may1
cooked: <p>Thank you for the answer and the example!</p>
<p>I expected model.half() could convert all the parameters and modules in the model into FP16.<br>
And h0, c0 are defined in that model.<br>
But they are also inputs to the LSTM layer. I didn’t convert h0, c0 into FP16 manually and that’s why I got the above error.</p>

username: ptrblck
cooked: <p>You are right that <code>model.half()</code> will transform all parameters and buffers to <code>float16</code>, but you also correctly mentioned that <code>h</code> and <code>c</code> are inputs. If you do not pass them explicitly to the model, it’ll be smart enough to initialize them in the right <code>dtype</code> for you in the <code>forward</code> method:</p>
<pre><code class="lang-python">model.half()
input = input.half()

out, (h, c) = model(input)
print(out.dtype, h.dtype, c.dtype)
</code></pre>

username: may1
cooked: <p>Thank you for the further information!</p>
<p>I had updated the model to have a if-condition flow to convert h0, c0 types before I saw your second comment.</p>
<p>But now I updated the model with your second suggestion, and yes it becomes smart enough to initialize them in the right dtype.</p>
<p>I was planning to trace the model and your answer helped me not to include condition flow.<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>