Quantization of a single tensor

username: kaoutar
cooked: <p>please, how to perform static quantization on a single tensor ? i want the steps to go from a float tensor to a quantized tensor (no math just pytorch code)</p>

username: jerryzh168
cooked: <p>do you have any larger context on what you are trying to do? we have a lot of quantize/dequantize ops that you can call but they may produce different tensors. e.g.</p>
<h3><a name="torchquantize_per_tensor-httpspytorchorgdocsstablegeneratedtorchquantize_per_tensorhtmltorch-quantize-per-tensor-1" class="anchor" href="#torchquantize_per_tensor-httpspytorchorgdocsstablegeneratedtorchquantize_per_tensorhtmltorch-quantize-per-tensor-1"></a>torch.quantize_per_tensor (<a href="https://pytorch.org/docs/stable/generated/torch.quantize_per_tensor.html#torch-quantize-per-tensor" class="inline-onebox" rel="noopener nofollow ugc">torch.quantize_per_tensor — PyTorch 2.1 documentation</a>):</h3>
<p>the old quantize op that gives you a quantized tensor in pytorch with quint8/qint8 etc. dtypes.</p>
<p>we are moving away from the above and wants to use native pytorch integer tensor directly:</p>
<h3><a name="torchopsquantized_decomposedquantize_per_tensor-httpsgithubcompytorchpytorchblobmaintorchaoquantizationfx_decomposedpyl37-2" class="anchor" href="#torchopsquantized_decomposedquantize_per_tensor-httpsgithubcompytorchpytorchblobmaintorchaoquantizationfx_decomposedpyl37-2"></a>torch.ops.quantized_decomposed.quantize_per_tensor (<a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/fx/_decomposed.py#L37" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/fx/_decomposed.py#L37</a>)</h3>
<p>This returns uint8/int8 pytorch tensor. and it doesn’t store scale/zero_point in the tensor itself.</p>

username: kaoutar
cooked: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> Thank you for replying, in fact i am far from model quantization and all this hassle of memory issues,</p>
<p>i have a ML task, the target is of float type, i want to do some experiments, one of them is convert the target to integer type and use LLM to predict an integer instead of a float, what i am looking for, is a function that helps me map float numbers to integers(quantize), and obviously another function to dequantize.</p>
<p>i’ve tried to read the quantization DOC on pytorch website, but i couldn’t , i was overwhelmed by the amount of text and details all talking about torch module.</p>
<p>any simple guide on how to do all types of quantization of a single tensor including how to find the best scale factor (i see there are a lot of techniques)</p>

username: jerryzh168
cooked: <p>quantization by itself is relatively straightforward, you can also just define your own quantize op: tensor / scale + zero_point.</p>
<p>maybe you can use some of our ops used in pytorch 2 export quantization: <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/fx/_decomposed.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/fx/_decomposed.py</a><br>
tests:<br>
<a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/core/test_quantized_tensor.py#L1463" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/test/quantization/core/test_quantized_tensor.py#L1463</a></p>
<p>to choose quantization parameters, you can also just call: <a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/core/test_quantized_tensor.py#L1515" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/test/quantization/core/test_quantized_tensor.py#L1515</a></p>