Question on quantize_per_channel and dequantize

mseeger: <p>I’d like to use low-level quantization of a tensor <code>x</code> of shape <code>(num_channels, blocksize)</code>. I know how to use <code>MovingAveragePerChannelMinMaxObserver</code> to obtain the quantisation params.</p>
<p>I also understand how to use <code>torch.quantize_per_channel</code> then, to get <code>q_x</code>. From the docs, the quant params see to somehow be attached to <code>q_x</code>.</p>
<p>But I’d now (for other reasons) like to store quantisation params separately from <code>q_x.int_repr()</code>.</p>
<p>Even though this is not documented anywhere, I suspect that <code>torch.dequantize</code> would do the right thing when applied to <code>q_x</code>. But in my case, I store <code>q_x.int_repr()</code> plus the quantisation params separately. What is a good way to dequantize then, given I have the internal representation and the parameters? How can I safely get a “quantized tensor” (i.e., like output of <code>quantize_per_channel</code>) from its parts?</p>

mseeger: <p>I found some issue pages with <code>torch._make_per_channel_quantized_tensor</code>:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/54056a58b25af18ef83de2464d8551199256feb510caaddf24c729e8762df6b1/pytorch/pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor" target="_blank" rel="noopener nofollow ugc">Introducing Quantized Tensor</a></h3>

  <p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>But I am not sure this is even callable. Why is this so hidden? Especially given that tensors with <code>torch.qint8</code> do not really implement many things, for example <code>torch.cat</code> fails for the per_channel variant.</p>

mseeger: <p>This actually works. I still wonder why this is not properly documented, and why I need to call an internal method to do something that seems pretty useful to me.</p>

mseeger: <p>Just a heads up. Quantization is not just useful for what your high-level stuff here uses it, namely quantize weights of big model. We also need it for inference, say to compress KV cache content. And this does need clean access to the low level primitives.</p>
<p>I wanted to use <code>bitsandbytes</code>, but found out this works on GPU only, and only on some, so I’ll implement the baseline with <code>torch</code> code. If you know a good way to get 4-bit quantization which always works (also on CPU), please let me know. I don’t want to do it myself, though.</p>

jerryzh168: <p>please checkout our new repo: <code>torchao</code> <a href="https://github.com/pytorch/ao" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/ao: PyTorch native quantization and sparsity for training and inference</a> that optimizes for LLM low precision inference,  training and finetuning (QAT)</p>

jerryzh168: <p>for CPU int4 weight only quant, you can check out this: <a href="https://discuss.pytorch.org/t/quantized-llm-inference-vs-quantized-matrix-multiplication-speed-in-cpu/217166/3" class="inline-onebox">Quantized LLM inference vs quantized matrix multiplication speed in CPU - #3 by jerryzh168</a></p>