About the int8 training question

UmeTFE: <p>Hi，</p>
<p>I know now PyTorch only supports float32 training. But I wonder if you have any idea or code example about int8 training, that is we use the int8 DNN model and retrain it in the int8 environment. There are one or two papers about int8 training, but can’t find the code.</p>
<p>Thanks in advance!</p>

J_Johnson: <p>Pytorch has int8 dtype compatibility. It’s referred to as “quantization”. See here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


UmeTFE: <p>Hi,</p>
<p>Thanks for your answer, I know PyTorch can fake quantize the DNN model to int8 in the end, but this training process is still in float32. But I want to know if PyTorch can or how to train the DNN with int8.</p>
<p>Best</p>

J_Johnson: <blockquote>
<h3><strong>Dynamic Quantization</strong></h3>
<p>The easiest method of quantization PyTorch supports is called <strong>dynamic quantization</strong>. This involves not just converting the weights to int8 - as happens in all quantization variants - but also converting the activations to int8 on the fly, just before doing the computation (hence “dynamic”). The computations will thus be performed using efficient int8 matrix multiplication and convolution implementations, resulting in faster compute. However, the activations are read and written to memory in floating point format.</p>
</blockquote>

UmeTFE: <p>Hi Johnson,</p>
<p>Thanks for your answer, I read the blog, but this blog, or “dynamic quantization”, seems like it still helps you to get an int8 model. But not train the model in an int8 environment. For pure int8 training, I think need to rewrite the BP and so on. I think this paper is what I want, " Towards Unified INT8 Training for Convolutional Neural Network", but I can’t find the code.</p>
<p>Thanks again for your answer! Have a nice day!</p>

J_Johnson: <p>I’m wondering if you read it or just skimmed it. The blog provides 3 methods for quantization. Only the last(3rd) method is “fake” quantization. There are two other methods in the blog, including dynamic quantization, which is not considered(at least by most) to be “fake” quantization.</p>

UmeTFE: <p>Yes, I tried the methods of quantization, but I don’t think they are the methods that I want. I want “training”, not “inference” or using float to simulate the “int”.</p>
<p>A similar question is here:</p><aside class="quote quote-modified" data-post="1" data-topic="82950">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/73ab20/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/problem-about-training-with-int8/82950">Problem about training with int8</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    Hello everyone, 
Recently, we are focusing on training with int8, not inference on int8. Considering the numerical limitation of int8, at first we keep all parameters in fp32 and only quantize convolution layer (conduct int8 operation) as it is the most compute-intensive part of a model. During the past months, we have achieved some progress (such accuracy comparable to fp32 training and faster than fp32 training), and our paper is accepted by CVPR2020 (<a href="https://arxiv.org/pdf/1912.12607.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1912.12607.pdf</a>). 
No…
  </blockquote>
</aside>


Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/umetfe">@UmeTFE</a> , the official PyTorch quantization APIs are geared towards inference.  They support QAT, but QAT is still training in floating point and converting to integer for inference.  You can check out torch.amp for training, but that only supports floating point types.</p>
<p>Doing training in the integer domain is not currently in core PyTorch, it’s more of an open research area as far as I know.  Would love to hear more about the motivation for this.</p>

srishti-git1110: <p>Hi <a class="mention" href="/u/umetfe">@UmeTFE</a><br>
Training the networks in int type is a bit tricky as the backpropagated gradients have a high probability of getting converted to 0 when in int mode.<br>
The most straight forward consequence of this is that this would lead the network to not learn at all or learn very slowly.</p>
<p>As far as I know, this is an active area of research, but still quantization is mostly used as in inference-only technique, and as <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a> mentioned PyTorch concurrently has support for three major quantization techniques - Dynamic quantization, PTSQ (Post training static quantization), and QAT (Quantization aware training).</p>
<p>With QAT, during the training we make the network aware of the fact that quantization shall be performed during inference. In other words, the network trains while being aware of it, and hence (hopefully) leads to the metrics to not worsen a lot after quantization is performed during inference.</p>

UmeTFE: <p>Hi,</p>
<p>Thank you so much for your reply.</p>
<p>I want to test the research idea ( which needs both training and inference processes )on the embedded system. But I don’t want to install PyTorch on it. That’s why I wonder if someone already did similar work before.</p>
<p>Thanks again for the answer.</p>

UmeTFE: <p>Hi,</p>
<p>Thanks for the answer, seems like now this question is still a research topic, and PyTorch is more caring about inference.</p>
<p>Thanks again for the answer and have a nice day!</p>

Vasiliy_Kuznetsov: <p><a class="mention" href="/u/umetfe">@UmeTFE</a> ,</p>
<p>Obviously I don’t have the full context of your problem, but usually people train in floating point and convert to integer for inference.  If your embedded environment supports fp32 or fp16, doing training in those dtypes should be easier (and probably supported by whichever framework you end up using) than trying to get integer training working.</p>

UmeTFE: <p>Hi,<br>
I agree with you, maybe int8 training is not a practical technique.</p>

srishti-git1110: <p><a class="mention" href="/u/umetfe">@UmeTFE</a> just curious if you’ve perhaps referred to/know of some literature around training NNs in integer precision?</p>

UmeTFE: <p>Hi,</p>
<p>There are a few papers about this, like the link I pasted above, but not so many…Most of the work is trying to use float to simulate int and decrease the “float to int” error.</p>
<p>I didn’t find any open-source code for int training. But I feel it’s too much work for one person to achieve the int training framework.</p>

2623448751: <p>Hi, I’m looking for training in INT8 too. I found that most the methods are about inference in INT8, like TensorRT. The nearest method I found is Pytorch Apex, but it didn’t support INT8 training. Have u find some methods that support training in INT8? If u have some ideas, pls contact me, thanks!</p>

HDCharles: <p>there are some efforts for training in Int8 (or some variant thereof) for improved speed, but in general the pytorch ao team doesn’t have tools to support that at the moment. We focus on taking trained models and optimizing them for performance with some tools oriented towards fine-tuning the accuracy of the quantized model so it better matches the original floating point model.</p>
<p>if you want more information about low precision training, i know Nvidia’s microscaling formats: <a href="https://arxiv.org/pdf/2310.10537" rel="noopener nofollow ugc">https://arxiv.org/pdf/2310.10537</a> paper has been getting a lot of attention.</p>