Converting model to torch.float16

HamidGadirov: <p>I converted my 3D training data to float16 for memory issues, but now there is an error:<br>
<code>RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.cuda.FloatTensor) should be the same</code><br>
How can I convert the model to <code>torch.float16</code> (or <code>torch.half</code>) before I start training?</p>
<p>The full error stack looks like this:</p>
<pre><code class="lang-auto">pred, info = model.update(imgs, gt, dataset, learning_rate, training=True)
  File "/home/hamid/Desktop/OpticalFlow/FlowSciVis/Flow-3D/model/RIFE.py", line 126, in update
    flow, mask, merged, flow_teacher, merged_teacher, loss_distill = self.flownet(torch.cat((imgs, gt), 1), scale=[1, 1, 1])
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 511, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hamid/Desktop/OpticalFlow/FlowSciVis/Flow-3D/model/IFNet.py", line 177, in forward
    flow, mask = stu[i](torch.cat((img0, img1), 1), None, scale=scale[i]) # stu[0]
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hamid/Desktop/OpticalFlow/FlowSciVis/Flow-3D/model/IFNet.py", line 94, in forward
    x = self.conv0(x)
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hamid/miniconda3/envs/gpu/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 567, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.cuda.FloatTensor) should be the same
</code></pre>

ptrblck: <p>If you want to use “pure” <code>float16</code> training, you would have to call <code>model.half()</code> to transform all parameters and buffers to <code>float16</code>, too.<br>
We generally recommend using <code>torch.cuda.amp</code> for mixed-precision training as it will be more stable than a pure <code>float16</code> training.</p>

HamidGadirov: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, that solved the previous issue. Now I am starting the training with with <code>torch.cuda.amp.autocast(True)</code>. I noticed however that the training takes more time per each iteration, could that be related to autocast mode? I am using NVIDIA TITAN V with CUDA Version 11.2.</p>

ptrblck: <p>This should not be the case. Could you post your model definition as well as the input shapes here, please?</p>

HamidGadirov: <p>I checked again and found that it is actually not the time per iteration that increased but the time between epochs. That is, the beginning of each new epoch takes increasingly more time and eventually slows down the training. Not sure why this happens, I have 3D volumetric data of size (128, 128, 128) with 4 channels (density and velocities) and I’m using the batch size of 45 at the moment, after I enabled the autocast mode.  My model consists of 3 blocks with several convolutional layers per each and PReLU activations. I recently added batch normalization to stabilize training after activations but my loss increases at some point at the beginning of the training and becomes <code>nan</code>, I’ll need to find why.</p>