How to apply per_tensor_symmetric activation quantization?

Wenlong_Shi: <p>I wish to perform quantization with a configuration in which both parameters and activations are quantized symmetrically. Here is my code:</p>
<pre><code class="lang-python">rn18 = models.resnet18().eval()
data = torch.randn(1, 3, 224, 224)
qconfig = torch.ao.quantization.QConfig(
    activation=torch.ao.quantization.observer.HistogramObserver.with_args(
        qscheme=torch.per_tensor_symmetric, dtype=torch.qint8
    ),
    weight=torch.quantization.default_per_channel_weight_observer,
)
qconfig_mapping = torch.ao.quantization.QConfigMapping().set_global(qconfig)
prepared = prepare_fx(rn18, qconfig_mapping, data)
for _ in range(10):
    prepared(data)
quantized_rn18 = convert_fx(prepared)
quantized_rn18.graph.print_tabular()
</code></pre>
<p>The printed result shows that the model is not quantized.<br>
How to solve the problem?</p>

supriyar: <p>You could use <code>torch.ao.quantization.default_symmetric_qnnpack_qconfig</code> as defined in <a href="https://github.com/pytorch/pytorch/blob/b8580b08976db89203f2ea7dda0f012520e9471a/torch/ao/quantization/qconfig.py#L65" class="inline-onebox" rel="noopener nofollow ugc">pytorch/qconfig.py at b8580b08976db89203f2ea7dda0f012520e9471a · pytorch/pytorch · GitHub</a></p>

Wenlong_Shi: <pre><code class="lang-auto">rn18 = torchvision.models.resnet18().eval()
data = torch.randn(1, 3, 224, 224)
qconfig = torch.ao.quantization.default_symmetric_qnnpack_qconfig
qconfig_mapping = torch.ao.quantization.QConfigMapping().set_global(qconfig)
prepared = prepare_fx(rn18, qconfig_mapping, data)
for _ in range(10):
    prepared(data)
quantized_rn18 = convert_fx(prepared)
quantized_rn18.graph.print_tabular()
</code></pre>
<p>I use the <code>default_symmetric_qnnpack_qconfig</code>, but the printed result shows that model is still not quantized.</p>

kmitsunami: <p>I have the same issue. Is there any workaround?</p>

HDCharles: <p>can you post a repro?</p>

kmitsunami: <p>Hi <a class="mention" href="/u/hdcharles">@HDCharles</a>,</p>
<p>Thanks for your reply. It’s basically same as <a class="mention" href="/u/wenlong_shi">@Wenlong_Shi</a>’s flow. You can reproduce the issue with below although my model is not resnet:</p>
<pre><code class="lang-auto">import os
import torchvision
import torch
from torch.ao.quantization import get_default_qconfig_mapping, default_symmetric_qnnpack_qconfig, get_default_qconfig
from torch.ao.quantization import QConfig
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx

# global_qconfig = get_default_qconfig("qnnpack") # this works
global_qconfig = default_symmetric_qnnpack_qconfig # this doesn't work
qconfig_mapping = get_default_qconfig_mapping("qnnpack")
qconfig_mapping.set_global(global_qconfig)

model = torchvision.models.resnet18().eval()
data = torch.randn(1, 3, 224, 224)
model_prepared = prepare_fx(model, qconfig_mapping, data)

for _ in range(10):
    model_prepared(data)

model_quantized = convert_fx(model_prepared)

def print_model_size(mdl):
    torch.save(mdl.state_dict(), "tmp.pt")
    print("%.2f MB" %(os.path.getsize("tmp.pt")/1e6))
    os.remove('tmp.pt')

print_model_size(model) # 46.83 MB
print_model_size(model_quantized) # 46.75 MB
</code></pre>
<p>The printed result shows that model is not quantized.</p>
<p>Thank you for your help.</p>

kmitsunami: <p>To add a little bit of background, what I would like to do is to match the sign of weights and activation (e.g. qint8/qint8). The qint8/quint8 combination such as get_default_qconfig(“qnnpack”) quantizes a model as expected, but when I set the sign to match, the quantization doesn’t work.</p>

kmitsunami: <p>Hi <a class="mention" href="/u/hdcharles">@HDCharles</a>,</p>
<p>Have you had a chance to check my code? I’d like to fix this issue and would appreciate your help.</p>
<p>Thanks.</p>