Torchscript with dynamic quantization produces inconsistent model outputs in Python and Java

westphal-jan: <p>Hello, I’ve been experimenting with torchscript and dynamic quantization and often have the issue that results of the models that are dynamically quantized are not consistent between Python and Java.<br>
To reproduce the issue I created a fork of the python java-demo: <a href="https://github.com/westphal-jan/java-demo" class="inline-onebox" rel="noopener nofollow ugc">GitHub - westphal-jan/java-demo</a>.</p>
<p>To setup you need to download libtorch and set its location in <code>build.gradle</code> (line 16).<br>
Download Link: <a href="https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.13.1%2Bcpu.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.13.1%2Bcpu.zip</a></p>
<p>I created a simple dummy model with one linear layer and export it unquantized and quantized here:  <code>create_dummy_models.py</code><br>
(The code can also be run using the dependencies defined in <code>requirements.txt</code> but I also commited the dummy models)</p>
<p>Python:</p>
<pre><code class="lang-auto">Unquantized model:
[[-2.758167028427124, 2.0038578510284424, -4.114053726196289, -1.2928203344345093, 1.4940322637557983]]
Quantized model:
[[-2.747678756713867, 1.9912285804748535, -4.110795021057129, -1.2891944646835327, 1.4982664585113525]]
</code></pre>
<p>You can run the java code with <code>./gradlew run</code>.<br>
Java:</p>
<pre><code class="lang-auto">Unquantized model:
data: [-2.758167, 2.0038579, -4.1140537, -1.2928203, 1.4940323]
[W qlinear_dynamic.cpp:239] Warning: Currently, qnnpack incorrectly ignores reduce_range when it is set to true; this may change in a future release. (function apply_dynamic_impl)
Quantized model:
data: [-2.7473624, 1.9966378, -4.110954, -1.283469, 1.4918814]
</code></pre>
<p>As you can see the output of the unquantized model is perfectly consistent while the output of the dynamically quantized model is slightly inconsistent. It might seem insignificant but with larger models like a transformer it becomes more obvious (differences usually already in the first decimal place). Am I misunderstanding something conceptually?</p>
<p>I thought as the code is compiled down to C++ and both examples run on the same architecture (CPU, x86_64) it should produce the same output even when using dynamic quantization (the activations are computed on the fly but they should still be deterministic).</p>
<p>Note: I made sure that Python and Java use the same version of Torch <code>1.13.1</code> which is the latest published mvn version (mvnrepository → org.pytorch/pytorch_java_only)</p>

jerryzh168: <p>I’m wondering if quantized engine is set to be the same in both cases? in python you can print it by: <code>print(torch.backends.quantized.engine)</code>, in java I’m not exactly sure how to do that though, but maybe you can look for similar apis</p>

westphal-jan: <p>Very interesting idea. I checked and set the quantization engine in python to qnnpack.<br>
<code>torch.backends.quantized.engine = "qnnpack"</code><br>
In this case, I get the same output as in Java. Even the warning message is included which was already hinting on <code>qnnpack</code>.</p>
<pre><code class="lang-auto">Unquantized model:
[[-2.758167028427124, 2.0038578510284424, -4.114053726196289, -1.2928203344345093, 1.4940322637557983]]
Quantized model:
&lt;repo-location&gt;/venv/lib/python3.8/site-packages/torch/nn/modules/module.py:1194: UserWarning: Currently, qnnpack incorrectly ignores reduce_range when it is set to true; this may change in a future release. (Triggered internally at ../aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp:239.)
  return forward_call(*input, **kwargs)
[[-2.7473623752593994, 1.9966378211975098, -4.1109538078308105, -1.2834689617156982, 1.4918813705444336]]
</code></pre>
<p>Therefore, it seems that Java uses <code>qnnpack</code> as default quantization engine. Do you know why this is the case for Java even though it’s running on x86_64? Is it because of libtorch?</p>

westphal-jan: <p>Or maybe you can help me with a different question. Why do these two backends produce different outputs at all?</p>

jerryzh168: <p>not exactly sure about this but I think it is related to compilation flags etc.</p>

jerryzh168: <p>the reason why two backends produce two different outputs is because in fbgemm, linear/conv are implemented with a special instruction that only works for 7 bit activations (<a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/core/test_quantized_op.py#L44" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/test/quantization/core/test_quantized_op.py#L44</a>), if you have 8 bit activation, it will have overflows, while qnnpack does not have this restriction.</p>

westphal-jan: <p>Thank you for the clarification. Is where a way to have the same behavior under both backends? I would rather have the same output and would be willing to lose a little bit of accuracy. Otherwise it is a hassle to keep track of model outputs for both backends.</p>

HDCharles: <p>you could just set the activation for qnnpack to 7 bits to get them to match that part, its unclear if they’ll match outputs exactly but that’d align them as much as possible. Generally you would accomplish this by using the fbgemm qconfigs for qnnpack for the ops you wish to align.</p>

westphal-jan: <p>From my experiments so far it is not possible to set the activation for <code>qnnpack</code> to 7 bits to match <code>fbgemm</code> using dynamic configs.<br>
First you would use the QConfig for the respective backend:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/qconfig.py#L249">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/qconfig.py#L249" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/qconfig.py#L249" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/ao/quantization/qconfig.py#L249</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="239" style="counter-reset: li-counter 238 ;">
          <li>        "backend: " + str(backend) +</li>
          <li>        f" not supported. backend must be one of {supported_backends}"</li>
          <li>    )</li>
          <li></li>
          <li>if version == 0:</li>
          <li>    if backend == 'fbgemm':</li>
          <li>        qconfig = QConfig(activation=HistogramObserver.with_args(reduce_range=True),</li>
          <li>                          weight=default_per_channel_weight_observer)</li>
          <li>    elif backend == 'qnnpack':</li>
          <li>        # TODO: make this compatible with xnnpack constraints</li>
          <li class="selected">        qconfig = QConfig(activation=HistogramObserver.with_args(reduce_range=False),</li>
          <li>                          weight=default_weight_observer)</li>
          <li>    elif backend == 'onednn':</li>
          <li>        if not torch.cpu._is_cpu_support_vnni():</li>
          <li>            warnings.warn(</li>
          <li>                "Default qconfig of oneDNN backend with reduce_range of false may have accuracy issues "</li>
          <li>                "on CPU without Vector Neural Network Instruction support.")</li>
          <li>        qconfig = QConfig(activation=HistogramObserver.with_args(reduce_range=False),</li>
          <li>                          weight=default_per_channel_weight_observer)</li>
          <li>    elif backend == 'x86':</li>
          <li>        qconfig = QConfig(activation=HistogramObserver.with_args(reduce_range=True),</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
The only difference being whether <code>reduce_range</code> is set to <code>true</code> or <code>false</code> for the activations. This makes sense regarding what we discussed so far.<br>
However, important is how this parameter is used during inference in the respective backend (which is the backend that is active when the model is loaded). Unfortunately, in the case of <code>qnnpack</code> the parameter <code>reduce_range</code> is not used which makes it impossible to match the behavior of <code>fbgemm</code>.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp#L247">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp#L247" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp#L247" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp#L247</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="237" style="counter-reset: li-counter 236 ;">
          <li>}</li>
          <li></li>
          <li>#endif // USE_FBGEMM</li>
          <li></li>
          <li>#ifdef USE_PYTORCH_QNNPACK</li>
          <li>template &lt;bool ReluFused&gt;</li>
          <li>at::Tensor PackedLinearWeightsQnnp::apply_dynamic_impl(</li>
          <li>    at::Tensor input,</li>
          <li>    bool reduce_range) {</li>
          <li>  if (reduce_range) {</li>
          <li class="selected">    TORCH_WARN_ONCE("Currently, qnnpack incorrectly ignores reduce_range when it is set to true; this may change in a future release.");</li>
          <li>  }</li>
          <li></li>
          <li>  using at::Tensor;</li>
          <li>  TORCH_CHECK(</li>
          <li>      input.dim() &gt;= 2,</li>
          <li>      "The dimension of input tensor should be larger than or equal to 2");</li>
          <li>  auto input_contig = input.contiguous();</li>
          <li>  // C(output) = A(input) x B(weight), where C, A, B are M x N, M x K, K x N</li>
          <li>  // matrices, respectively.</li>
          <li></li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


jerryzh168: <p>I see, not sure what is the status for qnnpack, whether it’s in maintenance or it’s still possible to add new support. cc <a class="mention" href="/u/kimishpatel">@kimishpatel</a> <a class="mention" href="/u/digantdesai">@digantdesai</a></p>