No Difference in Model size of BERT fine-tuned with amp and without amp

username: Ramesh_Kumar
cooked: <p>Hello Everyone,</p>
<p>I have fine-tuned <strong>bert-base model with amp</strong> and <strong>without amp</strong> using <code>MAX_SEQ_LEN=512</code>. I compared the performance among these models in terms of:</p>
<ol>
<li>Fine-tuning time</li>
<li>Inference time on CPU/GPU</li>
<li>Model size</li>
</ol>
<p>While conducting first experiment, I observed that in terms of <strong>Fine-tuning time</strong> , <code>bert model with amp</code> performs better as compare to <code>without amp</code>.</p>
<p>However, when I compare the inference time and model size, both models have same inference time and model size.</p>
<p>Could anyone please explain why this is the case?</p>

username: hx89
cooked: <p>Are you using PyTorch dynamic quantization for the model?</p>

username: ptrblck
cooked: <p>The model size regarding its parameters won’t be changed, as the operations (and intermediates) will be casted to FP16 for “safe” ops.<br>
So while you might be able to increase the batch size during training or inference, the <code>state_dict</code> won’t be smaller in size.</p>
<p>Which batch size are you using for inference? If you are seeing a speedup during training, you should also see it during inference. However, if your batch size is low (e.g. a single sample), the performance gain might be too small compared to the overheads of launching all kernels.</p>

username: Ramesh_Kumar
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your answer.</p>
<p>I have tried using different batch sizes e.g (8, 16, 64, 128). But I am not finding any difference.</p>
<p>Regarding the code, I am following examples here: <a href="https://pytorch.org/docs/stable/notes/amp_examples.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/amp_examples.html</a></p>
<p>Update: I am able to see the difference in inference time on <code>GPU</code> using</p>
<pre><code class="lang-auto">with autocast():
        with torch.no_grad():
            outputs = model(**inputs)
</code></pre>
<p>But when I compare the inference time on CPU, I do not notice any difference.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Ramesh_Kumar" data-post="4" data-topic="86708">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ramesh_kumar/48/12432_2.png" class="avatar"> Ramesh_Kumar:</div>
<blockquote>
<p>But when I compare the inference time on CPU, I do not notice any difference.</p>
</blockquote>
</aside>
<p>Automatic mixed precision is implemented for CUDA operations (and is thus in the <code>torch.cuda</code> namespace). By appying amp your GPU could use TensorCores for certain operations, which would yield a speedup. I don’t know if anything like that is implemented for CPU operations (and if I’m not mistaken not all operations are implemented for <code>HalfTensors</code> on the CPU).</p>

username: Ramesh_Kumar
cooked: <p>thanks for your response. No, I am not using dynamic quantization. But, since I cannot do mixed precision on cpu. Hence, I guess for CPU I have to switch to dynamic quantization. Am I right?</p>

username: Ramesh_Kumar
cooked: <p>So do you think, post-training quantization is better idea ? If we want to reduce inference time on cpu?</p>

username: ptrblck
cooked: <p>I’ve unfortunately never profiled the quantized models, so unsure what the expected speedup is.<br>
However, please let us know once if you are using the post-training quantized models and how large the performance gain is. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Ramesh_Kumar
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> just to give you update  regarding dynamic quantization on cpu. There is issue with quantized bert model which I and many others are facing. Here is the github issue link : <a href="https://github.com/huggingface/transformers/issues/2542" rel="nofollow noopener">https://github.com/huggingface/transformers/issues/2542</a></p>