Fundamental question on weight conversion fp32 to int8

username: APaul
cooked: <p>I had a basic question about quantization of a floating point number to int8 and would like to know the reason for difference between what I am computing.</p>
<p>For example if I have a floating point number 0.033074330538511, then to convert it to an int8 one, I used the following formula</p>
<pre><code class="lang-auto">quantized_weight = floor(float_weight.*(2^quant_bits))./(2^quant_bits)
</code></pre>
<p>Considering quant_bits as 8, the int8 value would be 0.031250000000000. But using pytorch quantization I am getting a value of 0.032944630831480</p>
<p>How can a int8 model have that much of precision in weight values?</p>
<p>To elaborate more I have an example model,</p>
<pre><code class="lang-auto">class M(nn.Module):

    def __init__(self):
            super(M, self).__init__()
            # QuantStub converts tensors from floating point to quantized
            self.quant = torch.quantization.QuantStub()
            self.conv = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=1)
            self.conv.weight = torch.nn.Parameter(torch.tensor([[[[ 0.03307433053851127625, -0.13484150171279907227, -0.21625524759292602539], 
                                                                    [ 0.14247404038906097412, -0.14247404038906097412, -0.24932956695556640625], 
                                                                    [ 0.32311078906059265137, -0.14501821994781494141, -0.21371106803417205811]]]]))
            self.conv.bias = torch.nn.Parameter(torch.tensor([0.1095]))
             # DeQuantStub converts tensors from quantized to floating point
            self.dequant = torch.quantization.DeQuantStub()

model_fp32 = M()
model_fp32.eval()
model_fp32.qconfig = torch.quantization.get_default_qconfig('fbgemm')
model_fp32_prepared = torch.quantization.prepare(model_fp32)
model_fp32_converted = torch.quantization.convert(model_fp32_prepared, inplace=True)
</code></pre>
<p>I didn’t pass any input through the model and when I printed the quantized model weights</p>
<pre><code class="lang-auto">model_fp32_converted.conv.weight()

tensor([[[[ 0.03294463083148002625, -0.13431271910667419434,
           -0.21540719270706176758],
          [ 0.14191532135009765625, -0.14191532135009765625,
           -0.24835182726383209229],
          [ 0.32184368371963500977, -0.14444953203201293945,
           -0.21287299692630767822]]]], size=(1, 1, 3, 3), dtype=torch.qint8,
       quantization_scheme=torch.per_channel_affine,
       scale=tensor([0.00253420230001211166], dtype=torch.float64),
       zero_point=tensor([0]), axis=0)
</code></pre>
<p>Although using this formula I can approximately compute the value</p>
<pre><code class="lang-auto">qunat_weight = floor(float_weight/scale)*scale
floor(0.03307433053851127625/0.00253420230001211166)*0.00253420230001211166
</code></pre>
<p>My query is</p>
<ul>
<li>How can a int8 save weight with that level of precision and how is pytorch quantization different from the method I showed?</li>
</ul>

username: ddang
cooked: <aside class="quote no-group" data-username="APaul" data-post="1" data-topic="148151">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ce7236/48.png" class="avatar"> APaul:</div>
<blockquote>
<p><code>0.03294463083148002625</code></p>
</blockquote>
</aside>
<p>I think this may be more related to floating point representation/arithmetic than quantization.</p>
<p>I don’t think you actually get the level of precision as what you are seeing when printing out the values in python.</p>
<p>This might be related:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.python.org/3/tutorial/floatingpoint.html">
  <header class="source">
      <img src="https://docs.python.org/3/_static/py.svg" class="site-icon" width="16" height="16">

      <a href="https://docs.python.org/3/tutorial/floatingpoint.html" target="_blank" rel="noopener nofollow ugc">Python documentation</a>
  </header>

  <article class="onebox-body">
    <img width="200" height="200" src="https://docs.python.org/3/_static/og-image.png" class="thumbnail onebox-avatar">

<h3><a href="https://docs.python.org/3/tutorial/floatingpoint.html" target="_blank" rel="noopener nofollow ugc">15. Floating Point Arithmetic:  Issues and Limitations</a></h3>

  <p>Floating-point numbers are represented in computer hardware as base 2 (binary) fractions. For example, the decimal fraction 0.625 has value 6/10 + 2/100 + 5/1000, and in the same way the binary fra...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/Machine_epsilon">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/Machine_epsilon" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://en.wikipedia.org/wiki/Machine_epsilon" target="_blank" rel="noopener nofollow ugc">Machine epsilon</a></h3>

<p>Machine epsilon or machine precision is an upper bound on the relative approximation error due to rounding in floating point arithmetic. This value characterizes computer arithmetic in the field of numerical analysis, and by extension in the subject of computational science. The quantity is also called macheps and it has the symbols Greek epsilon 
  
    
      
        ε
      
    
    {\displaystyle \varepsilon }
  
.
 There are two prevailing definitions. In numerical analysis, machine epsilo...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>