16 bit inference

erap129: <p>Hello,</p>
<p>I want to make inference at 16 bit precision (both for model parameters and input data). For example, I wish to convert numbers such as 1.123456789 to number with lower precision (1.123300000 for example)<br>
I wanted to ask if the following approach is correct for reducing the linear weights in a model to 16 bit:</p>
<pre><code class="lang-auto">    for layer in net_copy.modules():
        if type(layer) == nn.Linear:
            layer.weight = nn.Parameter(layer.weight.half().float())
</code></pre>
<p>Pytorch won’t work with half precision weights so I converted to half and back to float again. When I print the weights it’s hard for me to determine if this approach is behaving as I want it to, plus the performance of the network is identical before and after this transformation.</p>

ptrblck: <aside class="quote no-group" data-username="erap129" data-post="1" data-topic="87263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/838e76/48.png" class="avatar"> erap129:</div>
<blockquote>
<p>Pytorch won’t work with half precision weights</p>
</blockquote>
</aside>
<p>What kind of issue are you seeing? The current master and nightly binaries support mixed precision training as described <a href="https://pytorch.org/docs/master/amp.html">here</a>, so you might take a look at it, if you are using a GPU.</p>
<aside class="quote no-group" data-username="erap129" data-post="1" data-topic="87263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/838e76/48.png" class="avatar"> erap129:</div>
<blockquote>
<p>plus the performance of the network is identical before and after this transformation.</p>
</blockquote>
</aside>
<p>This is expected, as all operations would still be performed in FP32. FP16 operations, such as matrix multiplications and convolutions, can be accelerated on the GPU using TensorCores. If you just round the values, you wouldn’t get any performance gains.</p>

erap129: <p>I validated that the rounding is working properly by printing in binary, so I’m OK with that. I would just expect performance to deteriorate because of the lost precision. Maybe it is very very insignificant…</p>

ptrblck: <aside class="quote no-group" data-username="erap129" data-post="3" data-topic="87263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/838e76/48.png" class="avatar"> erap129:</div>
<blockquote>
<p>I would just expect performance to deteriorate because of the lost precision. Maybe it is very very insignificant…</p>
</blockquote>
</aside>
<p>I might have misunderstood what you meant by “performance”.<br>
Note that I was talking about the computation speedup using FP16 values on the GPU, while you seem to mean the model accuracy?</p>

erap129: <p>Yes, I’m not that concerned with speed.</p>

carter_Frank: <p>Hi, ptrblck，</p>
<p>Does pytorch support INT16 quantization-awre-training and static quantization, like INT8?</p>
<p>Thank you!</p>

ptrblck: <p>I don’t think so based on <a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/">this blog post</a> and these <a href="https://pytorch.org/docs/stable/quantization.html">docs</a>.</p>