Question: backend for quantization in pytorch

mahuichao: <p>Hello, guys<br>
recently I learned the source code of pytorch, I quantized my cnn layer and see the backend of it’s implementation. My system is Mac M1, so I can’t use GPU(CUDA), so I can only use CPU. From director y “ATen/native/quantized/cpu” I can see a lot of quantized layers like “qconv” and so on.<br>
In the file “qconv.cpp” I can see there are three macro: USE_FBGEM.<br>
MUSE_PYTORCH_QNNPACK.  AT_MKLDNN_ENABLED. Does this means that I can use anyone of these three? In another word, pytorch support three kind of quantization implementation in cpu and which one to use depends on what macro I defined ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f463edf6ca38456596ea74bff0f62737a6e0da80.png" data-download-href="https://discuss.pytorch.org/uploads/default/f463edf6ca38456596ea74bff0f62737a6e0da80" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f463edf6ca38456596ea74bff0f62737a6e0da80_2_690x230.png" alt="image" data-base62-sha1="yRYBfGW0IvqTgyAMrY1LuUtf8VW" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f463edf6ca38456596ea74bff0f62737a6e0da80_2_690x230.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f463edf6ca38456596ea74bff0f62737a6e0da80_2_1035x345.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f463edf6ca38456596ea74bff0f62737a6e0da80_2_1380x460.png 2x" data-dominant-color="E7E774"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2176×728 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thanks in advance.</p>

HDCharles: <p>yeah, there are different quantization engines, some info here: <a href="https://pytorch.org/docs/stable/quantization.html#note-for-native-cpu-backends" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a></p>
<p>and a code pointer here:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/5875a2fb3c4f21efec326dff6ba31d793a7194e1/torch/ao/quantization/qconfig.py#L221-L260">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/5875a2fb3c4f21efec326dff6ba31d793a7194e1/torch/ao/quantization/qconfig.py#L221-L260" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/5875a2fb3c4f21efec326dff6ba31d793a7194e1/torch/ao/quantization/qconfig.py#L221-L260" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/5875a2fb3c4f21efec326dff6ba31d793a7194e1/torch/ao/quantization/qconfig.py#L221-L260</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="221" style="counter-reset: li-counter 220 ;">
          <li>def get_default_qconfig(backend='x86', version=0):</li>
          <li>    """</li>
          <li>    Returns the default PTQ qconfig for the specified backend.</li>
          <li>
          </li>
<li>    Args:</li>
          <li>      * `backend` (str): a string representing the target backend. Currently supports</li>
          <li>        `x86` (default), `fbgemm`, `qnnpack` and `onednn`.</li>
          <li>
          </li>
<li>    Return:</li>
          <li>        qconfig</li>
          <li>    """</li>
          <li>    supported_backends = ["fbgemm", "x86", "qnnpack", "onednn"]</li>
          <li>    if backend not in supported_backends:</li>
          <li>        raise AssertionError(</li>
          <li>            "backend: " + str(backend) +</li>
          <li>            " not supported. backend must be one of {}".format(supported_backends)</li>
          <li>        )</li>
          <li>
          </li>
<li>    if version == 0:</li>
          <li>        if backend == 'fbgemm':</li>
      </ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/5875a2fb3c4f21efec326dff6ba31d793a7194e1/torch/ao/quantization/qconfig.py#L221-L260" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I think there’s also some xnnpack support, not 100% sure</p>

mahuichao: <p>Thanks guys, it helps a lot.</p>