Is there anything wrong with the Quantization Math?

ParGG: <p>Hi,</p>
<p>I have been working on quantization for a while and while working on a project in which I was planning to perform the more efficient bit-shift rescaling instead of a float multiplication, I have noticed some discrepancies with my mathematical model. After digging deeper in the implementation of my code, PyTorch and FBGEMM, I made some tests to check where the source of the error lies.</p>
<p>For the test I ran the following code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import random
import numpy as np
import pandas as pd

from collections import defaultdict
from matplotlib import pyplot as plt
from torch.quantization.observer import MinMaxObserver

SEED = 100
np.random.seed(SEED)
torch.manual_seed(SEED)
random.seed(SEED)

SHAPE_IN = 10
SHAPE_OUT = 10
BS = 100

MIN = -4
MAX = 4

class M(torch.nn.Module):
    def __init__(self):
        super(M, self).__init__()
        # QuantStub converts tensors from floating point to quantized
        self.quant = torch.quantization.QuantStub()
        self.fc = torch.nn.Linear(SHAPE_IN, SHAPE_OUT, bias=False)
        # DeQuantStub converts tensors from quantized to floating point
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, inp):
        # manually specify where tensors will be converted from floating
        # point to quantized in the quantized model
        inp = self.quant(inp)
        x = self.fc(inp)
        # manually specify where tensors will be converted from quantized
        # to floating point in the quantized model
        x = self.dequant(x)
        return x


def main():
    # create a model instance
    model_fp32 = M()

    # model must be set to eval mode for static quantization logic to work
    model_fp32.eval()

    # attach a global qconfig, which contains information about what kind
    # of observers to attach. Use 'fbgemm' for server inference and
    # 'qnnpack' for mobile inference. Other quantization configurations such
    # as selecting symmetric or assymetric quantization and MinMax or L2Norm
    # calibration techniques can be specified here.
    model_fp32.qconfig = torch.quantization.QConfig(
        activation=MinMaxObserver.with_args(dtype=torch.quint8),
        weight=MinMaxObserver.with_args(dtype=torch.qint8)
    )

    # Prepare the model for static quantization. This inserts observers in
    # the model that will observe activation tensors during calibration.
    model_fp32_prepared = torch.quantization.prepare(model_fp32)

    # calibrate the prepared model to determine quantization parameters for activations
    # in a real world setting, the calibration would be done with a representative dataset
    input_fp32 = torch.randn((BS, SHAPE_IN))
    model_fp32_prepared(input_fp32)

    # Convert the observed model to a quantized model. This does several things:
    # quantizes the weights, computes and stores the scale and bias value to be
    # used with each activation tensor, and replaces key operators with quantized
    # implementations.
    model_int8 = torch.quantization.convert(model_fp32_prepared)

    return model_int8

def test(val, model_int8):

    input_fp32 = torch.ones(
        (BS, SHAPE_IN)) * val # constant tensor with entries = val

    # compute quantized result with pytorch quantization
    pytorch_res = model_int8(input_fp32)

    return pytorch_res

def make_plot():
    d = defaultdict(list)
    model_int8 = main()
    for val in np.arange(MIN, MAX, 0.01):
        pyt_res = test(val, model_int8)
        d['val'].append(val)
        for i in range(SHAPE_OUT):
            d[f'pyt_res_{i}'].append(pyt_res[0][i].item())

    figsize = (1, 25)
    for i in range(SHAPE_OUT):
        plt.figure(figsize=figsize)
        df_a = pd.DataFrame(data={k: v for k, v in d.items() if k in ['val', f'pyt_res_{i}']})
        df_a.plot(x='val')
        plt.savefig(f"./experimental/res_{i}.png")
        plt.close()

if __name__ == '__main__':
    make_plot()
</code></pre>
<p>The operation performed here is basically a single matrix-vector product. Which was even simplified to have the inputs with all the same values. This means that the operation can be seen as:</p>
<pre><code class="lang-auto">Y = WX = W [val ,val, val]^T = torch.sum(W, dim= 1) * val
</code></pre>
<p>Example:</p>
<pre><code class="lang-auto">W = [[ 1, 10],
     [15,  7]|

X = [5, 5]^T

Y = WX = [11, 22]^T * 5 = [55, 110]^T
</code></pre>
<p>In my example what I am doing is the following, given a NN, I prepare it for quantization using 100 example datapoints. After converting it, I generate datapoints in the range [-4, 4] and probe the NN and check the output of ever feature.</p>
<p>EXPECTED BEHAVIOUR:<br>
Given the math, and the nature of the inputs (single value). The output of the network for each feature should be monotonic. This means it should look like a clamped ramp (either negative or positive depending on the weight).</p>
<p>RESULTS and OBSERVATIONS:<br>
To have an additional control, I checked also how TFLite behaves. My model and TF show for the same test the same behaviour. The output starts from the boundary of the range, when increasing the value of the input the output starts increasing/decreasing linearly until it reaches the other boundary of the range. So far so good.</p>
<p>PyTorch instead does not she the same behaviour. At least, it does not show that for <code>SHAPE_IN &gt; 1</code>. Here are some plots:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/081493fd3d74ed1c4c36201ebf016ead86a58379.png" alt="res_0" data-base62-sha1="19tUiKkKf7ZfUBcogzdQdybEnrz" width="640" height="480"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e65d31ac8a81902d22f3c4e0bb03ecf0c55193b.png" alt="res_3" data-base62-sha1="6Cs7yzTKEzsF7zAtPSi2h0mFM0z" width="640" height="480"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75f48b212882fc2bf2f3f7c98e2fba95014f9b8a.png" alt="res_9" data-base62-sha1="gPtNlU205w9Y3YfRMXqsmAhGUVs" width="640" height="480"></p>
<ul>
<li><strong>pyt_res_9</strong> is the 10th output feature and shows the correct trend (the slope could be positive as well)</li>
<li><strong>pyt_res_0</strong> is the 1st output feature and shows a change of slope at a certain point in the range</li>
<li><strong>pyt_res_3</strong> is the 4th output feature and shows twice a change of slope in the range</li>
</ul>
<p>What is going on? Is this a bug or is there some intended behaviour I am not aware about?</p>

jerryzh168: <aside class="quote no-group" data-username="ParGG" data-post="1" data-topic="155245">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pargg/48/50503_2.png" class="avatar"> ParGG:</div>
<blockquote>
<p>To have an additional control, I checked also how TFLite behaves. My model and TF show for the same test the same behaviour. The output starts from the boundary of the range, when increasing the value of the input the output starts increasing/decreasing linearly until it reaches the other boundary of the range. So far so good.</p>
</blockquote>
</aside>
<p>Sorry for the late reply, first of all, we do have tests for correctness of our quantized linear implementations: <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/core/test_quantized_op.py#L3465" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/test/quantization/core/test_quantized_op.py#L3465</a></p>
<p>For the problems you are seeing above, I’m not exactly sure if we would expect a monotonic increasing or decreasing trend. I think quantization itself is not a linear operation since it has clamping, also in calibration time you are calibration with torch.randn(((BS, SHAPE_IN)) and in test time you are testing with torch.ones(…) * val maybe that could be problematic as well.</p>

HDCharles: <p>thanks for including your code, the issue is your qengine</p>
<p>if you do:</p>
<p>torch.backends.quantized.engine=‘qnnpack’</p>
<p>this phenomenon goes away</p>
<p>if you do:</p>
<p>torch.backends.quantized.engine=‘fbgemm’ (which is the default qengine)</p>
<p>and in the observer (the activation one) use reduce_range=True</p>
<p>then the phenomenon should go away there too</p>
<p>basically the fbgemm quantized linear kernels have an overflow issue which causes the weird behavior you are seeing. This is why the default qconfig for fbgemm has reduce_range=True</p>

ParGG: <p>Dear <a class="mention" href="/u/jerryzh168">@jerryzh168</a> and <a class="mention" href="/u/hdcharles">@HDCharles</a>. Thank you very much for your replies. The problem was indeed in FBGEMM. When changing the backend to QNNPACK I could get the results to match across the different references.</p>
<p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> I am sorry I didn’t reply earlier to your post. I was writing a detailed reply that was taking longer than expected, but in the meantime <a class="mention" href="/u/hdcharles">@HDCharles</a> answered, and then checked right away if that was the solution to my problem.</p>
<p><a class="mention" href="/u/hdcharles">@HDCharles</a> thank you very much for the solution. Although, I cannot really understand where the error comes from. I check the C++ implementation of the Linear layer, quantization and so on, and couldn’t find the source of the overflow. Also, I could see that behaviour also when the input shape was 2, meaning that 2 multiplications and an accumulation were leading to overflow. But I cannot see how this can be possible given that the operations are performed in int32, and that specific operations required 17 bits to store the result in full precision (assuming zero_point = 0). I.e., <code>8bits multiplication -&gt; 16 bits, 16bits accumulation of 2 elements -&gt; 17 bits</code></p>
<p>Also, if due to an overflow, I would expect the plots to be discontinuous (127 + 1 = -128), on the other hand, the plots are continuous and there are changes in the slope.</p>
<p>Where does the overflow originate from? Is it an intended behaviour?</p>

HDCharles: <p>You’d have to ask the fbgemm developers, our team doesn’t do much with the kennels, we focus more on the model transformation piece.</p>
<p>If I were to guess though, I think it doesnt wrap when it overflows, it stays at that value. So if you have a<em>b+a</em>c, as a increases one of those terms reaches a maximum and then stays constant but the other keeps changing. This can fully explain the plots given the following 3 cases:</p>
<ol>
<li>
<p>If b<em>a and c</em>a are both saturated, the value stays constant</p>
</li>
<li>
<p>if one is saturated but b and c have the same sign, the slope gets shallower after the saturation point</p>
</li>
<li>
<p>if one is saturated but b and c have different signs then the first saturation point will be a local min/max.</p>
</li>
</ol>
<p>Although it accumulates to int32 I’m guessing the initial int8*int8 single element multiplies only accumulate to int15 which are then accumulated to int32 when added together. This is why setting reduce range=True solves the problem, since it removes a single bit from the range.</p>
<p>Note: take the above reasoning with a grain of salt since it’s just based on what I’m seeing with the plots with/without reduce range rather than an analysis of the kernels code.</p>

ParGG: <p>I think that’s a reasonable guess. Although, when having multiple terms, this should result in a higher number of local min/max, which I didn’t really notice. Maybe it is actually reflected in the lower number of noticeable quantization steps (This I could verify by my own).</p>
<p>Also, storing the result of an int8 multiplication in 15bits, seems a bit weird as a choice. Maybe is something in between, and they perform an FMA with 16bits (a*b + c).</p>
<p>I will ask the FBGEMM developers more about that and see if they answer.</p>
<p>Thank you very much again for your support and time.</p>

jerryzh168: <p>thanks <a class="mention" href="/u/hdcharles">@HDCharles</a> for debugging the issue. makes sense, reduce_range is needed because of the instruction that used to do matrix multiplication, here is more details: <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/core/test_quantized_op.py#L45" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_quantized_op.py at master · pytorch/pytorch · GitHub</a></p>