Fused 4-bit quantization support

Lewuathe: <p>Hi</p>
<p>I have found an issue closed to support 4-bit quantization although it’s not officially described in <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md" rel="noopener nofollow ugc">the doc</a>.</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/glow/issues/3463" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/glow</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/glow/issues/3463" target="_blank" rel="noopener nofollow ugc">[Quantization] Fused 4-bit quantization support</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-08-27" data-time="21:07:23" data-timezone="UTC">09:07PM - 27 Aug 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-01-22" data-time="20:32:03" data-timezone="UTC">08:32PM - 22 Jan 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/beicy" target="_blank" rel="noopener nofollow ugc">
          <img alt="beicy" src="https://avatars.githubusercontent.com/u/37386895?v=4" class="onebox-avatar-inline" width="20" height="20">
          beicy
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">This issue is created to track fused 4-bit quantization support in Glow.
The fused 4-bit quantization tensor is represented as follows:
| .......</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>model-compiler seems not to support <code>Int4</code> quantization.</p>
<pre><code class="lang-auto">model-compiler: for the -quantization-precision option: Cannot find option named 'Int4'!
</code></pre>
<p>We want to try this quantization mode. It is available in the latest Glow? How can we use that?</p>

jfix: <p>Hi, I believe <code>-quantization-precision</code> only supports <code>Int8</code> and <code>Int16</code> right now. For 4-bit quantization, it’s currently only supported by a few ops, such as <a href="https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html" rel="noopener nofollow ugc"><code>EmbeddingBag</code></a> and <a href="https://caffe2.ai/docs/operators-catalogue.html#sparselengthssum" rel="noopener nofollow ugc"><code>SparseLengthsSum</code></a> given that these ops often are loading from extremely large embedding tables that can be shrunk significantly using 4-bit quantization.</p>
<p>If you wanted to use it for other operators we’d need to expand its support across a variety of different operators.</p>

Lewuathe: <p>Thank you for the information. I wanted to try 4-bit quantization if possible to compare the accuracy and performance of models generated by Glow.</p>
<p>For the case of when we want to do 4-bit quantization for some operators like <code>EmbeddingBag</code> and <code>SparseLengthSum</code>, how can we enable that? <code>-quantization-precision</code> does not support the <code>Int4</code> for now.</p>
<p>Is that automatically enabled?</p>

jfix: <p>So right now I don’t think we have automatic Glow-based quantization support for this. We have pre-quantized Glow kernels for executing these ops, but they are only ever loaded pre-quantized from the input model, i.e. they are quantized in PyTorch, Caffe2, etc. before Glow ever loads them.</p>
<p>In order to support this we’d need to extend the Glow profiler to (1) support per-row profiling, and then (2) use that per-row profiling to do the 4-bit quantization.</p>
<p>(Note that our 4bit quantization support for EmbeddingBag and SparseLengthsSum are both rowwise quantized)</p>

Lewuathe: <p>Got it. Thank you for the tailored explanation!</p>