How to inference with smoothquant quantized model with pytorch?

oleotiger: <p><a href="https://github.com/mit-han-lab/smoothquant/" rel="noopener nofollow ugc">SmoothQuant</a> can only run on GPUs with Cutlass support.</p>
<p>There are many models quantized with SmoothQuant on Hugging Face. I want to run inference with them on an ARM CPU-only server, performance notwithstanding. I have struggled for a long time but haven’t found a viable way to achieve this.</p>
<ul>
<li>Should I modify SmoothQuant or Torch-int?</li>
<li>PyTorch supports quantization with QNNPACK, and it provides both module (e.g., quantized_linear, with unspecified input tensor datatype) and functional interfaces (e.g., linear, with input as quint8 and weight as qint8). These do not match the interface with Cutlass in Torch-int.</li>
</ul>
<p>I would greatly appreciate it if someone could provide thoughts, workflows, or example code/pseudo code.</p>

HDCharles: <p>Unless i am mistaken, smoothquant is just input-weight equalization applied to a quantized linear with dynamic quantization of some kind on the activation. Let me know if you want something different.</p>
<p>we have input-weight equalization for fx quantization</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_equalize_fx.py#L344">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_equalize_fx.py#L344" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_equalize_fx.py#L344" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/test/quantization/fx/test_equalize_fx.py#L344</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="334" style="counter-reset: li-counter 333 ;">
          <li>    }</li>
          <li></li>
          <li>    m = TestBranchingWithEqualizationModel().eval()</li>
          <li>    example_inputs = (torch.randn(1, 5),)</li>
          <li>    prepared = prepare_fx(</li>
          <li>        m, specific_qconfig_dict, example_inputs=example_inputs,</li>
          <li>        _equalization_config=default_equalization_qconfig_dict)</li>
          <li>    self.checkGraphModuleNodes(prepared, expected_node_occurrence=eq_branching_node_occurrence)</li>
          <li></li>
          <li>@skipIfNoFBGEMM</li>
          <li class="selected">def test_input_weight_equalization_convert(self):</li>
          <li>    """ Tests that the modified model for equalization (before quantization)</li>
          <li>    returns the same output as the original model</li>
          <li>    """</li>
          <li></li>
          <li>    tests = [(SingleLayerLinearModel, 2), (LinearAddModel, 2), (TwoLayerLinearModel, 2),</li>
          <li>             (SingleLayerFunctionalLinearModel, 2), (FunctionalLinearAddModel, 2),</li>
          <li>             (TwoLayerFunctionalLinearModel, 2),</li>
          <li>             (LinearReluModel, 2), (LinearReluLinearModel, 2), (LinearReluAddModel, 2),</li>
          <li>             (FunctionalLinearReluModel, 2), (FunctionalLinearReluLinearModel, 2),</li>
          <li>             (ConvModel, 4), (TwoLayerConvModel, 4), (SingleLayerFunctionalConvModel, 4),</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>and we have dynamic quant with fx<br>
<a href="https://pytorch.org/docs/stable/quantization.html#prototype-fx-graph-mode-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a> which looks like it works for qnnpack. I think you’re stuck with per-tensor quantizaiton on the activation though.</p>
<p>those two are intended to compose so let us know if that doesn’t work.</p>
<p>also i think you’re looking at the wrong quantized linear op, what you’d want is <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/dynamic/modules/linear.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/dynamic/modules/linear.py</a></p>

oleotiger: <p>Thank you for your responses to both of my questions.</p>
<p>Actually, what I’m interested in is solely the inference of LLM with PyTorch. There are many quantized LLM models with SmoothQuant on Hugging Face (e.g., <a href="https://huggingface.co/mit-han-lab/opt-125m-smoothquant/tree/main" rel="noopener nofollow ugc">opt-125m-smoothquant</a>). I want to run a test on one or a few of them on an ARM CPU-only platform. But, I believe I haven’t found the correct method yet.</p>
<p>Here’s what I’ve tried: I wanted to run the model with <a href="https://github.com/mit-han-lab/smoothquant" rel="noopener nofollow ugc">SmoothQuant</a>, but it calls Torch-int, which is built on Cutlass. I wanted to replace all quantization interfaces on Torch-int or SmoothQuant, but found that quantized linear in Torch-int supports qint8 for activation. I only found quint8 for activation in the PyTorch backend.</p>
<p>I haven’t found the correct location to eliminate Cutlass while also supporting the correct interface in PyTorch.</p>

HDCharles: <p>When you say you want to do smoothquant do you mean something you can define mathematically or a particular repo?</p>
<p>Smoothquant is a technique from <a href="https://arxiv.org/pdf/2211.10438.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2211.10438.pdf</a> that can be reproduced as outlined above on cpu or from a variety of repos on cuda.</p>
<p>the Repo <a href="https://github.com/mit-han-lab/smoothquant" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mit-han-lab/smoothquant: [ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models</a> is geared towards cuda as far as i can tell, if you want to use that on cpu you’re kind of fitting a round peg in a square hole and that’s reflected in the interface difference you’re seeing. I also think the function you’re looking at isn’t a drop in replacement. Its doing static quantization rather than dynamic quantization that smoothquant needs (mathematically)</p>

oleotiger: <p>What I’m interested in is a particular repository, as mentioned above - the <a href="https://github.com/mit-han-lab/smoothquant" rel="noopener nofollow ugc">SmoothQuant repository based on CUDA</a>. However, instead of focusing on the quantization of a model, what I’m really concerned with is how to run inference with an already SmoothQuant-quantized model on an Arm CPU server. I discovered that I can only replace the Cutlass interface that SmoothQuant calls through torch-int, such as linear_a8_w8_bfp32_ofp32. Therefore, my course of action should be to either find a replacement for the interface in QNNPACK or implement them by myself. Is this correct?</p>

HDCharles: <p>it sounds like your question is less about quantization and more about running a model from an external repo on cpu. I can speculate but you’re probably better off asking them directly.</p>
<p>as mentioned above, qnnpack does have a dynamically quantized linear op, but it uses affine per tensor activation quantization and per-channel weight quantization.</p>

chenster_liu: <p>I think <a href="https://github.com/intel/neural-compressor/blob/master/docs/source/smooth_quant.md" rel="noopener nofollow ugc">this</a> is what you are looking for. Intel’s neural compressor integrates smoothquant and I’ve tested in my CPU-only machine.</p>