Is the AMX accelerator used on Apple silicon?

Question:
From issue #47702 on the PyTorch repository, it is not yet clear whether PyTorch already uses AMX on Apple silicon to accelerate computations. It might do this because it relies on the operating system’s BLAS library, which is Accelerate on macOS. For reasons not described here, Apple has released little documentation on the AMX ever since its debut in the A13 chip.
If PyTorch does already use AMX, then that is ~1.3 TFLOPS of processing power. For comparison, the M1 GPU has 2.6 TFLOPS. The issue linked above was raised partially because PyTorch lacked hardware acceleration on Apple devices for a very long time. If AMX is in fact used and has comparable performance to GPU acceleration, then many people might want to know.
Could anyone investigate whether the AMX is being used? You may need to learn a bit of Swift, which provides direct access to Accelerate and microsecond-level precision for profiling. Note that M1 has one AMX, while M1 Pro/Max has two. Here are some helpful links for anyone who wishes to investigate this:

Swift-Colab repository 4 (has links to Swift crash course)
Swift for PyTorch demo 18

Context:
Some more helpful links:

Explanation of AMX 22
PythonKit repository 8


[SOLVED] No torch version in MacOS Catalina

Question:
I’m trying to restore all python packages on MacOS 10.15.7 Catalina and now I faced the following error:
ERROR: Could not find a version that satisfies the requirement torch==1.10.0 (from versions: none)
ERROR: No matching distribution found for torch==1.10.0
I’ve tried to point Pip to the page with -f parameter. Is there any way to install it to Catalina?
MacOS 10.15.7 Catalina
Python 3.10.1

Context: Ok, I’ve figured it out. I had to downgrade python to 3.9.9 so it has nothing to do with MacOS.


Import SummaryWriter gives AttributeError: AttributeError: module ‘setuptools._distutils’ has no attribute ‘version’

Question:
I’m using PyTorch on a MacBook with an Apple M1 processor. I am using miniforge to install ARM PyTorch and ARM Tensorflow.
As far as I can tell, everything works except for Tensorboard. For instance, if I refer to the tutorial at torch.utils.tensorboard — PyTorch 1.10.1 documentation 9 I am stopped at the third line:
from torch.utils.tensorboard import SummaryWriter
raises an error:
 AttributeError: module 'setuptools._distutils' has no attribute 'version'.
These are the versions I’m using:
python                    3.9.9
tensorboard               2.6.0
pytorch                   1.10.0
tensorflow-deps           2.7.0
tensorflow-estimator      2.7.0
tensorflow-macos          2.7.0
tensorflow-metal          0.3.0

Is this a known issue? And is there a workaround?

Context: Yes, this was a known issue which was already fixed in this PR 126. You could thus either update PyTorch to the nightly release or downgrade setuptools==59.5.0 as a workaround.


Question: Hey there, I just wanted to ask if there is any support planned for the M1 MacBook chips and if so, if they will utilize the GPU cores or the Neural Cores of the M1 processors. Is there any public info about this?

Context: As someone also interested this, I believe this long running issue is the defacto public info on M1 support: GPU acceleration for Apple's M1 chip? · Issue #47702 · pytorch/pytorch · GitHub 63


Question:
hello , friends
i new with PyTorch
my correct laptop is 2018 mac book pro  with Graphics  card of
“Intel Iris Plus Graphics 655 1536 MB”
when i use “easy OCR” the result are accurate but it take like 8 seconds to get the result
for my project i need fast result
i want to ask you : if i will buy the new mac book pro late 2021
with m1 pro chips (MacBook Pro 14-inch and 16-inch - Technical Specifications - Apple (UK))
will it give me fast  result in “easy OCR” ?

Context:
my project is to build a scrabble bot
(scrabble game is playing on the screen)
and the bot is player 1 and there is 3 other human player
so he need in real time to recognizing and processing text on the screen


Question:
Hi,
I am installing pytorch locally on MacOs El Capitan 10.11.6 in a conda environment.  In attempts at both pip3 install torch torchvision and the
conda install pytorch torchvision -c pytorch, I haven’t been able to import torch (“module not found”).  Would anybody know what I am doing wrong and what is the best way to fix it?  Thanks.

Context:
Hi,
You want to make sure that the “pip3” you’re using is matching the “python” that you are using. In particular, make sure that they are both for python3.
To make sure you use the right pip to match the command line python3, you can do python3 -m pip the_rest_of_your_pip_command.


Error: “trainloader” and “writer” are not defined, when it has been defined, in tutorial on viz model via tensorboard

Question:
I’m following the PyTorch official tutorial on how to viz model via TensorBoard 1 in my notebook here 2.
(Note that I copied the exact code from the tutorial webpage)

However, I continued to get the following error on the section to write image to tensorboard, although obviously “trainloader” has been defined in the first cell.

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-6d95240dcdaa> in <module>
      1 # get some random training images
----> 2 dataiter = iter(trainloader)
      3 images, labels = dataiter.next()
      4
      5 # create grid of images

NameError: name 'trainloader' is not defined

After I re-ran the first cell to create the trainloader twice, the error points to the writer object, which obviously has been created in previous cell.

NameError: name 'writer' is not defined
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-5-6d95240dcdaa> in <module>
     10
     11 # write to tensorboard
---> 12 writer.add_image('four_fashion_mnist_images', img_grid)
NameError: name 'writer' is not defined

Appreciate if you can advise what could have caused this problem.

Context:
Based on the errors I guess your Jupyter notebook might have trouble executing the cells somehow?
Are you seeing the same issues when you run the script from a terminal (outside of a notebook)?


Implement inference of quantized network with primitive operations

Question:
I want to implement quantized network in pure C. One of the purposes is to get full understanding on how the operations with quantized tensors work.
I made PQT with Renset-18 architecture and got good accuracy with fbgemm backend.
Now I am struggling to replicate the operations. I decided that the simplest to start is addition in Renset block.
self.skip_add = nn.quantized.FloatFunctional()

And during inference I can add two tensors via
out1 = self.skip_add.add(x1, x2)

where x1 and x2 are tensors of torch.Tensor type, quantized with fbgemm backend during post training quantization procedure.
I expected out2_int = x1.int_repr() + x2.int_repr() should be the same as out1.int_repr() (with probably need of clamping in the needed range). However that is not the case.
Can anyone please provide me with any information on how to implement operations with quantized tensors?
Below I dump the example outputs.
print(x1)

    ...,
      [-0.0596, -0.0496, -0.1390,  ..., -0.0596, -0.0695, -0.0099],
      [-0.0893,  0.0000, -0.0695,  ...,  0.0596, -0.0893, -0.0298],
      [-0.1092,  0.0099,  0.0000,  ..., -0.0397, -0.0794, -0.0199]]]],
   size=(1, 256, 14, 14), dtype=torch.quint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.009925744496285915,
   zero_point=75)

print(x2)

      ...,
      [ 0.1390, -0.1669, -0.0278,  ..., -0.2225, -0.0556, -0.1112],
      [ 0.0000, -0.1669, -0.0556,  ...,  0.0556,  0.1112, -0.2781],
      [ 0.1390,  0.1669,  0.0278,  ...,  0.2225,  0.4171,  0.0834]]]],
   size=(1, 256, 14, 14), dtype=torch.quint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.02780967578291893,
   zero_point=61)

print(x1.int_repr())

      ...,
      [69, 70, 61,  ..., 69, 68, 74],
      [66, 75, 68,  ..., 81, 66, 72],
      [64, 76, 75,  ..., 71, 67, 73]]]], dtype=torch.uint8)

print(x2.int_repr())

      ...,
      [66, 55, 60,  ..., 53, 59, 57],
      [61, 55, 59,  ..., 63, 65, 51],
      [66, 67, 62,  ..., 69, 76, 64]]]], dtype=torch.uint8)

print(self.skip_add.add(x1, x2))

      ...,
      [ 0.0904, -0.2109, -0.1808,  ..., -0.2712, -0.1205, -0.1205],
      [-0.0904, -0.1808, -0.1205,  ...,  0.1205,  0.0301, -0.3013],
      [ 0.0301,  0.1808,  0.0301,  ...,  0.1808,  0.3314,  0.0603]]]],
   size=(1, 256, 14, 14), dtype=torch.quint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.03012925386428833,
   zero_point=56)

print(self.skip_add.add(x1, x2).int_repr())

      ...,
      [59, 49, 50,  ..., 47, 52, 52],
      [53, 50, 52,  ..., 60, 57, 46],
      [57, 62, 57,  ..., 62, 67, 58]]]], dtype=torch.uint8)

 print(x1.int_repr() + x2.int_repr())

      [135, 125, 121,  ..., 122, 127, 131],
      [127, 130, 127,  ..., 144, 131, 123],
      [130, 143, 137,  ..., 140, 143, 137]]]], dtype=torch.uint8)

Context:
It looks to me like its just dequantizing, adding together, and then requantizing:
import torch

x = torch.randn(10,10)
y = torch.randn(10,10)

#arbitrary scales and zero_points
zp_x = 1
zp_y = 2
zp_z = 3

s_x = .1
s_y = .2
s_z = .3

#quantize tensors
xq = torch.quantize_per_tensor(x, s_x, zp_x, torch.qint8)
yq = torch.quantize_per_tensor(y, s_y, zp_y, torch.qint8)

#setup add operation
add_op = torch.nn.quantized.QFunctional()
add_op.scale = s_z
add_op.zero_point = zp_z

zq_qfunc = add_op.add(xq,yq)
print("QFunctional output", zq_qfunc)

#manually do operation
xdq = xq.dequantize()
ydq = yq.dequantize()

#add together
zdq = xdq+ydq

#requantize
zq_manual = torch.quantize_per_tensor(zdq, s_z, zp_z, torch.qint8)

print("Manual output", zq_manual)

print("difference", zq_manual.int_repr()-zq_qfunc.int_repr())

You can get the specifics about the quant/dequant process formulas here: Quantization API Reference — PyTorch master documentation


Why model size is reduced in Dynamic Quantization?

Question:
In the tutorials of Quantization, there is a mention that the model size would be reduced by using Dynamic Quantization(DQ).
After I merged the DQ codes, I found the model size is reduces(5MB->2MB). that what I expected.
However, I am wondering why the model is reduces.
so, I tried to log the model state_dict and the log is following.

the original model’s state_dict()
				'model.layers.3.residual_group.blocks.5.mlp.fc1.weight', tensor([[-0.0133, -0.0458, -0.0438,  ..., -0.0109,  0.0203, -0.0292],
				[ 0.0185,  0.0241,  0.0071,  ...,  0.0204,  0.0048, -0.0240],
				[-0.0027, -0.0198, -0.0116,  ..., -0.0246, -0.0079, -0.0145],
				...,
				[-0.0086,  0.0161,  0.0068,  ...,  0.0200,  0.0013, -0.0164],
				[ 0.0080, -0.0006, -0.0074,  ...,  0.0420, -0.0109,  0.0062],
				[-0.0169,  0.0129,  0.0252,  ..., -0.0208, -0.0016, -0.0064]])), ('model.layers.3.residual_group.blocks.5.mlp.fc1.bias', tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])),

this is DQ’s state_dict()
				('model.layers.3.residual_group.blocks.5.mlp.fc1.scale', tensor(1.)),

				('model.layers.3.residual_group.blocks.5.mlp.fc1.zero_point', tensor(0)),

				('model.layers.3.residual_group.blocks.5.mlp.fc1._packed_params.dtype', torch.qint8),

				('model.layers.3.residual_group.blocks.5.mlp.fc1._packed_params._packed_params', (tensor([[-0.0131, -0.0459, -0.0435,  ..., -0.0107,  0.0203, -0.0292],
				[ 0.0185,  0.0238,  0.0072,  ...,  0.0203,  0.0048, -0.0238],
				[-0.0024, -0.0197, -0.0119,  ..., -0.0244, -0.0078, -0.0143],
				...,
				[-0.0083,  0.0161,  0.0066,  ...,  0.0197,  0.0012, -0.0161],
				[ 0.0078, -0.0006, -0.0072,  ...,  0.0417, -0.0107,  0.0060],
				[-0.0167,  0.0131,  0.0250,  ..., -0.0209, -0.0018, -0.0066]],
			   size=(120, 60), dtype=torch.qint8,
			   quantization_scheme=torch.per_tensor_affine, scale=0.0005962323630228639,
			   zero_point=0), tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
				0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
			   requires_grad=True))),

I think DQ’s weight/bias looks like fp32 not int8.
in this reason, I wonder how to reduce model size by using fp32 weight format.
could you tell me why the this is happend?
Thank you for your help.

Context:
The answer is in your question.
fp32 is a floating point expressed by 32bits.
qint8 is a quantized integer expressed by 8bits.
In the aspect of computer science, the expected size of model would be 1.25MB but not 2MB.
However, there is additional information which contains a quantization scheme and not all the tensors inside the model can be converted into qint8.
For more info, please read the docs Quantization — PyTorch 1.10.1 documentation


Question:
When I quantize a model with “Hardswish”, It’s got a Error with info:
AttributeError: ‘Hardswish’ object has no attribute ‘activation_post_process’,
And my coonfig is like this:

image1745×361 47.8 KB

How could I fix this error? thanks

Context:
you are using qat prepare but normal qconfig. Also in the mapping, nnq.Hardswish isn’t a qat module. If you are intending to do qat you should do something like
    import torch
    qconfig = torch.quantization.get_default_qat_qconfig("fbgemm")
    model = torch.nn.Sequential(torch.nn.modules.Hardswish(), torch.nn.modules.Linear(1,1))
    model.qconfig = qconfig
    model_prep = torch.quantization.prepare_qat(model)
    print(model_prep)
    model_prep(torch.randn(1,1))

if you are not intending to do qat you should do something like
    import torch
    qconfig = torch.quantization.get_default_qconfig("fbgemm")
    model = torch.nn.Sequential(torch.nn.modules.Hardswish(), torch.nn.modules.Linear(1,1))
    model.qconfig = qconfig
    model_prep = torch.quantization.prepare(model)
    print(model_prep)
    model_prep(torch.randn(1,1))


Question:
I found that in some case, the pytorch result of qat linear will be error. Next I will describe it.
I insert some print in model forword code

print(layers_3_out)
layers_4_out = layers_4(layers_3_out)
print(layers_4._packed_params)
print(layers_4_out)

The layers_4 is define in init:

self.layers_4 = nn.Linear(in_features=16, out_features=1, bias=True)

So as above shows, I print the input and linear params and output, and the result is as follows:

tensor([[5.0954, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.6091,
0.0000, 1.3409, 2.6818, 4.2908, 2.1454, 2.9500, 1.8772],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[5.2295, 1.4750, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.3409,
0.0000, 1.4750, 3.0840, 4.4249, 2.0113, 2.9500, 1.7432],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[4.9613, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.7432],
[4.9613, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.7432],
[4.9613, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.7432],
[4.6931, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.4136, 3.8886, 1.8772, 2.8159, 1.7432],
[4.6931, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.4136, 3.8886, 1.8772, 2.8159, 1.7432],
[4.6931, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.4136, 3.8886, 1.8772, 2.8159, 1.7432],
[4.6931, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.4136, 3.8886, 1.8772, 2.8159, 1.7432],
[4.4249, 1.0727, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.2068, 2.2795, 3.6204, 1.8772, 2.8159, 1.7432],
[4.4249, 1.0727, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.2068, 2.1454, 3.7545, 1.7432, 2.8159, 1.7432],
[4.5590, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.2068, 2.2795, 3.7545, 1.7432, 2.8159, 1.7432],
[4.4249, 1.0727, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.2068, 2.1454, 3.7545, 1.7432, 2.8159, 1.7432],
[4.4249, 1.0727, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.2068, 2.1454, 3.7545, 1.7432, 2.8159, 1.7432],
[4.4249, 1.0727, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.2068, 2.1454, 3.7545, 1.7432, 2.8159, 1.7432],
[4.6931, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.4136, 4.0227, 1.7432, 2.8159, 1.7432],
[4.4249, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.2068,
0.0000, 1.3409, 2.4136, 3.8886, 1.7432, 2.6818, 1.7432],
[4.5590, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.2068,
0.0000, 1.3409, 2.4136, 3.8886, 1.7432, 2.8159, 1.6091],
[4.4249, 1.0727, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.2068, 2.1454, 3.7545, 1.7432, 2.8159, 1.7432],
[4.6931, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.4136, 4.0227, 1.7432, 2.8159, 1.7432],
[4.6931, 1.2068, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.4136, 3.8886, 1.8772, 2.8159, 1.7432],
[4.9613, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.7432],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772],
[5.3636, 1.4750, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.4750, 3.0840, 4.5590, 2.1454, 2.9500, 1.8772],
[5.0954, 1.3409, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.4750,
0.0000, 1.3409, 2.6818, 4.2908, 2.0113, 2.9500, 1.8772]],
size=(32, 16), dtype=torch.quint8,
quantization_scheme=torch.per_tensor_affine, scale=0.13408887386322021,
zero_point=173)
(tensor([[-0.5172, -0.6572,  0.2640,  0.4471,  0.2855, -0.5549,  0.4741,  0.4364,
-0.5118, -0.0162, -0.3448, -0.5172, -0.6896, -0.3448, -0.6465, -0.4902]],
size=(1, 16), dtype=torch.qint8,
quantization_scheme=torch.per_tensor_affine, scale=0.005387257784605026,
zero_point=0), tensor([-0.1708], requires_grad=True))
tensor([[3.7054],
[3.8143],
[3.8143],
[3.8143],
[3.8143],
[3.8143],
[3.5964],
[3.8143],
[3.8143],
[3.8143],
[3.8143],
[3.9233],
[3.9233],
[3.9233],
[3.9233],
[4.0323],
[4.1413],
[4.0323],
[4.1413],
[4.1413],
[4.1413],
[3.9233],
[4.1413],
[4.1413],
[4.1413],
[3.9233],
[3.9233],
[3.8143],
[3.8143],
[3.8143],
[3.5964],
[3.8143]], size=(32, 1), dtype=torch.quint8,
quantization_scheme=torch.per_tensor_affine, scale=0.10898102819919586,
zero_point=186)

Then I calculate the first result of this linear, as

Expand Table

input
5.0954
1.2068
0
0
0
0
0
0
1.6091
0
1.3409
2.6818
4.2908
2.1454
2.95
1.8772
bias

weight
-0.5172
-0.6572
0.264
0.4471
0.2855
-0.5549
0.4741
0.4364
-0.5118
-0.0162
-0.3448
-0.5172
-0.6896
-0.3448
-0.6465
-0.4902
-0.1708

mul
-2.63534088
-0.79310896
0
0
0
0
0
0
-0.82353738
0
-0.46234232
-1.38702696
-2.95893568
-0.73973392
-1.907175
-0.92020344
-12.79820454

And I found that the result should be -12.79820454, but the result from pytorch is  3.7054, it is error.
And before inference, I qat and prepare model as before:

net.train()
net.qconfig = torch.quantization.get_default_qat_qconfig(“qnnpack”)
net.fuse_modules()
torch.quantization.prepare_qat(net, inplace=True)
net.load_state_dict(new_state_dict, strict=True)
net.eval()
net.apply(torch.quantization.disable_observer)
net = torch.quantization.convert(net)

And In this model, other linear is calculated correct, except this one. So if anyone meet this problem before, Can anyone help me?

Context:
The Code as Follow:
import torch
import torch.nn as nn
import torch.optim as optim

class mturn_predictor(nn.Module):
    def __init__(self, *args):
        super().__init__()
        self.QuantStub = torch.quantization.QuantStub()
        self.cls_score_parking_goal_combine_multi_layers_0_new = nn.Linear(in_features=16, out_features=1, bias=True)
        self.DeQuantStub = torch.quantization.DeQuantStub()

    def forward(self, x):
        Cat_out = self.QuantStub(x)
        cls_score_parking_goal_combine_multi_layers_0_new = self.cls_score_parking_goal_combine_multi_layers_0_new(Cat_out)
        DeQuantStub_out = self.DeQuantStub(cls_score_parking_goal_combine_multi_layers_0_new)
        return DeQuantStub_out

def generate_input():
    example = torch.tensor([[4.4249,0,0,0,2.5477,0,4.559,0,0,0,0.5364,3.3522,2.1454,0,0,0]])
    return example

if __name__ == "__main__":
    model = mturn_predictor()
    input_data = generate_input()

    print(input_data)
    output_data = model(input_data)

    ## convert into qat model
    model.train()
    model.qconfig = torch.quantization.get_default_qat_qconfig("qnnpack")
    torch.quantization.prepare_qat(model, inplace=True)

    output_data = model(input_data)

    model.eval()
    model_qat = torch.quantization.convert(model)
    print("torch.quantization.convert Done")

    weight_tensor = torch.tensor([[-0.5172,-0.6572,0.264,0.4471,0.2855,-0.5549,0.4741,0.4364,-0.5118,-0.0162,-0.3448,-0.5172,-0.6896,-0.3448,-0.6465,-0.4902]])
    weight_qat = torch.quantize_per_tensor(weight_tensor, 0.00538725778460502, 0, torch.qint8)

    bias_tensor = torch.tensor([-0.17078383266925812])
    model_qat.cls_score_parking_goal_combine_multi_layers_0_new._packed_params.set_weight_bias(weight_qat, bias_tensor)
    model_qat.cls_score_parking_goal_combine_multi_layers_0_new.scale = 0.10898102819919586
    model_qat.cls_score_parking_goal_combine_multi_layers_0_new.zero_point = 186

    model_qat.QuantStub.register_buffer('scale', torch.tensor([0.13408887386322021]))
    model_qat.QuantStub.register_buffer('zero_point', torch.tensor([173], dtype=torch.long))

    print(model_qat)
    #import pdb; pdb.set_trace()

    print("Input:", input_data)
    print("Weight:", model_qat.cls_score_parking_goal_combine_multi_layers_0_new._packed_params)
    output_data = model_qat(input_data)
    print("Output:", output_data)

    print("Matmul: ", torch.matmul(input_data, torch.transpose(weight_tensor, 0, 1)))

The result of Matmul and Linear is different,
Output: tensor([[7.4107]])
Matmul:  tensor([[-2.7979]])


SlowFast-Architecture on FPGA - Conversion from .pkl-File to .pth-File

Question:
Hello,
does anybody know how to convert a model given as .pkl-file into .pth?
in my master thesis I’m working on a solution to deploy a SlowFast-Network for human action recognition on an embedded platform with an FPGA. I use the open source codebase PySlowFast and have tested the inference on GPU already (just to see how it works). My embedded platform has an FPGA from Xilinx and I want to use Vitis AI from Xilinx for the quantization and compilation of the model.
The Vitis AI - quantizer needs a pre-trained PyTorch model, generally a .pth-file and a float model definition as input. The SlowFast-Network is saved as a .pkl-file. Does anybody know how I can get a .pth-file and a float model definition from this .pkl-file respectively from the PySlowFast codebase?
For example detectron offers a python skript to convert a model (.yaml and .pkl) to a model.pb. Do you maybe know a similar solution for PyTorch-Models?
Many thanks!

Context:
Hello,
I would be very sceptical of defining format from the extension. I myself saw lot’s of different ways of saving pytorch models of the same format with different extensions (though pytorch’s documentations suggests conventinal way).
Not 100% sure but it may end up that .pth and .pkl are refering to the same format.


Question:
Hi, I tried quantization but the cpu I have uses avx and not avx2. So I get this error:
RuntimeError: Didn't find engine for operation quantized::linear_prepack NoQEngine

with this code:
model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

Is there any way to use quantization with a cpu that only supports avx?
Sincerely

Context: You could try to use the qnnpack engine with SSE4_1, but “AVX not AVX2” is generally not terribly well-suported.


Inference error after int8 quantization with pytorch

Question:
when I inferenced my model with int8 quantization， I meet the following error: what should I do to solve it?
NotImplementedError: Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build).

Context:
Can you provide the model code which you are trying to quantize. FYI quantization is not implemented yet for CUDA
At the moment PyTorch doesn’t provide quantized operator implementations on CUDA - this is the direction for future work. Move the model to CPU in order to test the quantized functionality.


Slower ops in quantized::mul, quantized::cat

Question:
Hello,
During quantization, I realized quantized operations such as quantized::mul, quantized::cat are x10 slower than fp32 ops.
Is the only workaround wrapping those functions with dequant() and quant()? Please refer to the profiling below…
FP32 Profiling (CPU)
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                            Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
        aten::mkldnn_convolution        36.28%      14.237ms        37.56%      14.741ms       1.340ms       7.88 Mb           0 b            11
        aten::upsample_nearest2d         6.71%       2.632ms         8.38%       3.288ms     469.671us      15.33 Mb      15.32 Mb             7
                       aten::mul         6.10%       2.395ms         6.10%       2.395ms     342.071us      19.41 Mb      19.41 Mb             7
                      aten::_cat         5.54%       2.175ms         6.69%       2.626ms     375.100us      19.41 Mb           0 b             7
                      aten::_cat         5.28%       2.074ms         6.45%       2.533ms     361.814us      19.41 Mb           0 b             7
        aten::upsample_nearest2d         4.50%       1.766ms         7.26%       2.851ms     407.229us      15.33 Mb      15.32 Mb             7

Quantized Profiling (CPU)
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                   quantized::mul        20.94%      21.950ms        21.79%      22.839ms       3.263ms       4.85 Mb           0 b             7
                   quantized::cat        18.42%      19.306ms        18.84%      19.741ms       2.820ms       4.85 Mb           0 b             7
                   quantized::cat        17.84%      18.692ms        18.27%      19.143ms       2.735ms       4.85 Mb           0 b             7
                quantized::conv2d         8.18%       8.576ms         8.86%       9.285ms       1.326ms       1.02 Mb      -4.08 Mb             7
          quantized::batch_norm2d         4.65%       4.878ms         5.34%       5.598ms     933.033us     980.00 Kb      -7.75 Kb             6
                quantized::conv2d         4.35%       4.561ms         4.90%       5.134ms     733.500us     981.00 Kb      -3.83 Mb             7
                   quantized::mul         4.34%       4.553ms         5.13%       5.380ms     768.571us       1.02 Mb           0 b             7
            quantized::leaky_relu         1.73%       1.810ms         2.13%       2.234ms     372.417us     980.00 Kb           0 b		6

Context:
yeah I think a lot of them are using dequant/quant to simulate the quantized operation for quantized::cat, but for quantized::mul I remember we do have more efficient implementations in fbgemm/qnnpack. Wondering which quantized engine are you using right now? and which platform did you run this?
you can print quantized engine by: print(torch.backends.quantized.engine)


Fuse Modules - How to fuse modules in the Sequential block

Question:
I am trying to fuse some modules in the AlexNet
model = torchvision.models.alexnet()

with the architecture:
AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace=True)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace=True)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)

This is the code I use to fuse Conv2d and ReLU
torch.quantization.fuse_modules(model, [["features[0]", "features[1]"]])

or
torch.quantization.fuse_modules(model, [["features0", "features.1"]])

but got the same error
AttributeError: 'AlexNet' object has no attribute 'features[0]'
How can I fuse modules within a Sequential block?

Context:
Hi @lhkhiem28 , [["features.0", "features.1"]] should work.  More generally, you could print out the fully qualified name of every module in the model with something like
for name, mod in model.named_modules():
  print(name, mod)


Is it planned to support nn.Embeddings quantization?

Question:
First of all, I would like to thank you for the awesome  torch.quantization . But at the moment, the quantization of embeddings is not supported, although ususally it’s one of the biggest (in terms of size) parts of the model (in NLP).
I tried to use  nn.Embeddings  as  nn.Linear  because they have a very similar nature, but get the following error:
RuntimeError: Could not run 'aten::index_select' with arguments from the 'QuantizedCPUTensorId' backend. 'aten::index_select' is only available for these backends: [CPUTensorId, CUDATensorId, SparseCPUTensorId, SparseCUDATensorId, VariableTensorId].

So I’m interested whether it’s planned to support  nn.Embeddings  quantization?

Context:
It is possible to set the qconfig of Embeddings to None if you wish to skip quantizing them. For example
class EmbeddingWithLinear(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.emb = torch.nn.Embedding(num_embeddings=10, embedding_dim=12)
                self.fc = torch.nn.Linear(5, 5)
                self.emb.qconfig = None
                self.qconfig = default_qconfig

            def forward(self, indices, linear_in):
                return self.emb(indices), self.fc(linear_in)


How to extract the quantized weight of quantized NN model

Question:
I am using Post Training Quantization and try to extract the quantized weight for inference phase, but I failed.
I try to directly use
for weight in quantized_model.state_dict():
np.set_printoptions(suppress=True)
print(weight, “\n”, quantized_model.state_dict()[weight].detach().cpu().clone().numpy())
get “TypeError: NumPy conversion for QuantizedCPUQInt8Type is not supported”
Could give me any advice for extracting the quantized weight from the quantized model?
Thank you very much!!

Context: the error message shows that the weight is already a quantized weight, it’s just numpy operation is not supported in quantized weight, so if you remove the numpy() call it should work.


How to load quantized model for inference

Question:
hello,
I have Resnet classification from torchvision and I finetune it for my custom dataset and save my model(final_model.pth)
know I want to quantize this model and it quantizes without a problem and if I inference write away it is ok but if I load my quantize model and then inference it has Error
######Quantization Aware Training#######
import torch
model = torch.load("final_model.pth", map_location='cpu')
model.train()
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
#model_fp32_fused = torch.quantization.fuse_modules(model,[['conv1', 'bn1', 'relu']])
model_fp32_prepared = torch.quantization.prepare_qat(model)
train_annotations_dir = val_annotations_dir = "/content/annotations.json"
train_img_dir = val_img_dir = "/content/images"
epoch = 2
batch_size = 32
image_size = 640
model_path=model_fp32_prepared
gpu_num = 0
learning_rate =0.01
opt = "adam"
momentum = 0.9
patience = 5
output = "/content/output2/"
#
m = run_classification_train(train_annotations_dir,train_img_dir,val_annotations_dir,val_img_dir,epoch,batch_size,image_size,model_path,gpu_num,learning_rate,momentum,opt,patience)
m.to("cpu")
model_int8 = torch.quantization.convert(m)
final_model_path = m
torch.save(model_int8.state_dict(), "state_qmodel.pth")
torch.save(model_int8, "fm.pth")
inference(val_annotations_dir,val_img_dir,image_size,batch_size,final_model_path,output)

if I load state_qmodel.pth  I get ERROR

Error(s) in loading state_dict for ResNet:
Unexpected key(s) in state_dict: “conv1.bias”, “conv1.scale”, “conv1.zero_point”, “layer1.0.conv1.bias”, “layer1.0.conv1.scale”, “layer1.0.conv1.zero_point”, “layer1.0.conv2.bias”, “layer1.0.conv2.scale”, “layer1.0.conv2.zero_point”, “layer1.0.conv3.bias”, “layer1.0.conv3.scale”, “layer1.0.conv3.zero_point”, “layer1.0.downsample.0.bias”, “layer1.0.downsample.0.scale”, “layer1.0.downsample.0.zero_point”, “layer1.1.conv1.bias”, “layer1.1.conv1.scale”, “layer1.1.conv1.zero_point”, “layer1.1.conv2.bias”, “layer1.1.conv2.scale”, “layer1.1.conv2.zero_point”, “layer1.1.conv3.bias”, “layer1.1.conv3.scale”, “layer1.1.conv3.zero_point”, “layer1.2.conv1.bias”, “layer1.2.conv1.scale”, “layer1.2.conv1.zero_point”, “layer1.2.conv2.bias”, “layer1.2.conv2.scale”, “layer1.2.conv2.zero_point”, “layer1.2.conv3.bias”, “layer1.2.conv3.scale”, “layer1.2.conv3.zero_point”, “layer2.0.conv1.bias”, “layer2.0.conv1.scale”, “layer2.0.conv1.zero_point”, “layer2.0.conv2.bias”, “layer2.0.conv2.scale”, “layer2.0.conv2.zero_point”, “layer2.0.conv3.bias”, “layer2.0.conv3.scale”, “layer2.0.conv3.zero_point”, “layer2.0.downsample.0.bias”, “layer2.0.downsample.0.scale”, “layer2.0.downsample.0.zero_point”, “layer2.1.conv1.bias”, “layer2.1.conv1.scale”, “layer2.1.conv1.zero_point”, “layer2.1.conv2.bias”, “layer2.1.conv2.scale”, “layer2.1.conv2.zero_point”, “layer2.1.conv3.bias”, “layer2.1.conv3.scale”, “layer2.1.conv3.zero_point”, “layer2.2.conv1.bias”, “layer2.2.conv1.scale”, "layer2.2.conv1.zero_point

import torch
model = torch.load("/content/final_model.pth")
final_model_path = model.load_state_dict(torch.load('/content/state_qmodel.pth'))
inference(val_annotations_dir,val_img_dir,image_size,batch_size,final_model_path,output)

Context:
Hi @m.safari, when you run the quantization APIs it changes the state dict, because quantized layers can have different fields compared to their floating point counterparts.  Therefore, when you load a quantized checkpoint, the recommendation is to create the fp32 architecture, run the quantization APIs (on random weights), and then load the quantized state dict.  In your example, it would be something like
# create fp32 model
model = torch.load("/content/final_model.pth")

# quantize it without calibration (weights will not be final)
model.train()
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
#model_fp32_fused = torch.quantization.fuse_modules(model,[['conv1', 'bn1', 'relu']])
model_fp32_prepared = torch.quantization.prepare_qat(model)
model_int8 = torch.quantization.convert(m)

# load the real state dict
model_int8.load_state_dict(torch.load('/content/state_qmodel.pth')


Question:
class TemporalShift(nn.Module):
    def __init__(self, n_segment=3, n_div=8):
        super(TemporalShift, self).__init__()
        self.n_segment = n_segment
        self.fold_div = n_div

    def forward(self, x):
        x = self.shift(x, self.n_segment, fold_div=self.fold_div)
        return x

    @staticmethod
    def shift(x, n_segment, fold_div=3):
        nt, c, h, w = x.size()
        n_batch = nt // n_segment
        x = x.view(n_batch, n_segment, c, h, w)

        fold = c // fold_div

        #out = torch.zeros_like(x)
        out = x.clone().zero_()
        out[:, :-1, :fold] = x[:, 1:, :fold]  # shift left
        out[:, 1:, fold: 2 * fold] = x[:, :-1, fold: 2 * fold]  # shift right
        out[:, :, 2 * fold:] = x[:, :, 2 * fold:]  # not shift

        return out.view(nt, c, h, w)

when using fx to tracing tsm, i get this error:
out[:, :-1, :fold] = x[:, 1:, :fold]
TypeError: ‘Proxy’ object does not support item assignment

Context:
Hi @keyky, this is a limitation of symbolic tracing with FX.  Here is a workaround using torch.fx.wrap:
@torch.fx.wrap
def shift_left(out, x, fold):
    out[:, :-1, :fold] = x[:, 1:, :fold]  # shift left

@torch.fx.wrap
def shift_right(out, x, fold):
    out[:, 1:, fold: 2 * fold] = x[:, :-1, fold: 2 * fold]  # shift right

@torch.fx.wrap
def not_shift(out, x, fold):
    out[:, :, 2 * fold:] = x[:, :, 2 * fold:]  # not shift

class TemporalShift(nn.Module):
    def __init__(self, n_segment=3, n_div=8):
        super(TemporalShift, self).__init__()
        self.n_segment = n_segment
        self.fold_div = n_div

    def forward(self, x):
        x = self.shift(x, self.n_segment, fold_div=self.fold_div)
        return x

    @staticmethod
    def shift(x, n_segment, fold_div=3):
        nt, c, h, w = x.size()
        n_batch = nt // n_segment
        x = x.view(n_batch, n_segment, c, h, w)

        fold = c // fold_div

        #out = torch.zeros_like(x)
        out = x.clone().zero_()
        shift_left(out, x, fold)
        shift_right(out, x, fold)
        not_shift(out, x, fold)

        return out.view(nt, c, h, w)

m = TemporalShift()
ms = torch.fx.symbolic_trace(m)


Input type disagreed with the model type

Question:
I quantized both my model and input to “int8”, but an error generated,

Screenshot from 2021-12-22 16-23-051631×328 31.8 KB

why does the model require that input type is float

Context: Hi， I’ve found the error and remove the “normalize”, it worked.


Question:
return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors)
RuntimeError: “compute_indices_weights_linear” not implemented for ‘Half’，
pytorch1.9.1 did not support float16？

Context: ok, I have known where the problem is. I should add the quant(x) into the forward headmost.


Question:
Hi,
I’m trying to perform QAT on GPT2 model, but I’m a bit confused about the documentation regarding the QuantStub.

Where should I place the QuantStub and DeQuantStub? Based on my understanding I should place the first QuantStub after the Embedding layer and the DequantStub after the Relu activation layer of the FFN; then subsequently the QuantStub will be after the previous DequantStub, which is before the second linear layer of the FFN of the previous decoder layer. Is that correct?

I can only fuse the first linear layer and the Relu activation layer of the FFN in each of the decoder layers. Am I right?

Decoder-Only-Architecture-used-by-GPT-2722×852 62.6 KB

Thanks in advance!
JM

Context: I am also having problems using Quantization-Aware with GPT-2, did you find a solution? Can you share it with me? Thank you.


Question: if my model consists of deeper and nested layers, should I  insert quant&dequant into every layer?

Context:
If all the operators in the model can be quantized, you can insert a quant and dequant at the beginning and end of the model. See (beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.10.0+cu102 documentation 1 for more details.
If there are only specific layers that can be quantized then you would have to wrap them individually using quant-dequant blocks in eager mode. Alternatively, you could try the FX graph mode quantization flow which should automate the process - (prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 1.10.0+cu102 documentation 1


Why does quantized tensor * 2 returns error?

Question:
I just recently figured out this simple operation returns error.
class QNet(nn.Module):
    def __init__(self):
        super(QNet, self).__init__()
        self.quant = QuantStub()
        self.dequant = DeQuantStub()
        self.conv = nn.Conv2d(1, 1, 1)
        self.bn = nn.BatchNorm2d(1)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = x * 2           # <<<< Error
        x = self.bn(x)
        x = self.relu(x)
        x = self.dequant(x)
        return x

NotImplementedError: Could not run 'aten::empty_strided' with arguments from the 'QuantizedCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty_strided' is only available for these backends: [CPU, CUDA, Meta, BackendSelect, Python, Named, Conjugate, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].

Is this because quantized tensor only operates with quantized objects?

Context: please take a look at the response here: NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'QuantizedCPU' backend - #3 by jerryzh168 3


RuntimeError: Overflow when unpacking long, during FX mode calibration

Question:
Hello, I am following FX mode post training static quantization tutorial 1, and got an error when running the last line of this code below.
# 1. FX Mode quantization
    model_to_quantize = copy.deepcopy(model_fp)
    model_to_quantize.eval()
    qconfig_dict = {"": torch.quantization.get_default_qconfig('qnnpack')}

    # prepare (Insert observer)
    model_prepared = quantize_fx.prepare_fx(model_to_quantize, qconfig_dict)

    # calibrate
    model_prepared.eval()
    model_prepared(img, mask)

-----------------------------------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\dalsee\PycharmProjects\PConv\mymodel.py", line 266, in <module>
    model_prepared(img, mask)
  File "C:\Users\dalsee\Anaconda3\envs\deeplearning\lib\site-packages\torch\fx\graph_module.py", line 512, in wrapped_call
    raise e.with_traceback(None)
RuntimeError: Overflow when unpacking long

img and mask are both the input of the model, which is 256x256 tensor converted from graylevel picture.

Context:
Hi @dalseeroh, did you make any changes when running the tutorial locally? If yes, can you share them here?
cc @jerryzh168 in case we need to update the tutorial.


Quantized Batch norm parameters not saved in state_dict

Question:
I found out that the scale and zero point parameters of the BatchNorm2d module after conversion with post-training quantization are not included the model.state_dict(). Therefore, when saving the state dict of the converted model to a file, these parameters are lost. This yields different results when doing inference just after post-training quantization, or whether first saving the state dict of the quantized model to a file and later on loading it again.
This does not happen when the batch norm module is fused with another module (like convolution), because then, because the scale and zero point is saved for the module if it is fused (e.g., with convolution module).
An alternative is to save the state dict of the model before running torch.quantization.convert(model). However, is there a specific reason why the scale and zero point of the quantized batch norm module is not included in the state dict of the model after conversion?
Reference: torch.nn.quantized.modules.batchnorm — PyTorch 1.10.0 documentation (here you see that scale and zero point are taken from the activation_post_process.calculate_qparams(), but when loading a new model without doing proper calibration, these values are not set correctly)

Context: I think this might be a bug, thanks for reporting, opened an issue here: quantized batchnorm parameters/buffers not saved in state_dict · Issue #69808 · pytorch/pytorch · GitHub 3


Question:
I am trying to quantize a model, but I got an error when I executes:
model_int = torch.quantization.convert(model_fp_prepared)
print("Conversion completed")
q_output = model_int(img, mask)

NotImplementedError: Could not run ‘aten::empty.memory_format’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit Internal Login 1 for possible resolutions. ‘aten::empty.memory_format’ is only available for these backends: [CPU, CUDA, Meta, MkldnnCPU, SparseCPU, SparseCUDA, BackendSelect, Python, Named, Conjugate, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].

It looks like it happened here:
output = self.input_conv(input_x * mask)

Context:
I guess I figured it out. This error seems to happen when I try to multiply those quantized tensors(input_x, mask). The workaround I took is:
# First, dequantize the quantized tensor
 input_x = self.dequant(input_x)
 mask = self.dequant(mask)
# Do the operation and quantize it back
 masked = input_x * mask
 masked = self.quant(masked)

 input_x = self.quant(input_x)
 mask = self.quant(mask)

 output = self.input_conv(masked)

Seems like pretty tedious work but it works. However, can I use self.quant() multiple times like that? or Should I use self.quant1(), self.quant2(), self.quant3() separately?


Static quantization/QAT change shape of embedding layers

Question:
Hello,
I am trying to apply different kinds of quantization (static, dynamic and quantization-aware trainings) to a BERT model taken from the transformers library. My model works as expected with no quantization or with dynamic quantization, but both static quantization and quantization-aware training return a strange error which makes it seem as though QuantStub and DeQuantStub are changing tensor shapes.
Here is a simplified version, the comments indicate where the errors are and the reasoning for adding the various tweaks. I included the two error stacks described in the comments (error A and B) at the end this post.
def __init__(self, ...):
    ...
    self.model = AutoModelForTokenClassification.from_pretrained('bert-base-multilingual-cased', config=...)
    ...

def train(self, trainset, devset, num_epochs, **kwargs):
    self.trainer = Trainer(model=self.model, args=..., train_dataset=trainset, eval_dataset=devset)

    # Without any kind of quantization, the model works as expected

    # Case 1: dynamic quantization for Linear layers, works as expected
    if self.quantization == "dynamic":
        self.trainer.train()
        self.model = quant.quantize_dynamic(
            self.model, {nn.Linear}, dtype=getattr(torch, self.quant_type)
        )
        self.trainer.model = self.model

    # Case 2: quantization-aware training. The errors are exactly the same as for static quantization.
    elif self.quantization == "qat":
        self.model.quant = quant.QuantStub()
        self.model.dequant = quant.DeQuantStub()

            # This snippet is necessary in the first place because of https://discuss.pytorch.org/t/89154, otherwise I get a "AssertionError: The only supported dtype for nnq.Embedding is torch.quint8"
            if isinstance(module, nn.Embedding):
                # Both alternatives here cause different errors
                module.qconfig = None # Causes Error A (presumably because of operating on both quantized and non-quantized tensors)
                module.qconfig = quant.float_qparams_dynamic_qconfig # Causes Error B (why?)

        self.model.qconfig = quant.get_default_qat_qconfig('fbgemm')
        quant.prepare_qat(self.model, inplace=True)

        Trainer.compute_loss = quant_compute_loss
        Trainer.prediction_step = quant_prediction_step
        self.model.train()
        self.trainer.train()
        self.model.eval()

        self.model = quant.convert(self.model, inplace=True)
        self.trainer.model = self.model

    # Case 3: static quantization, not included here. The code is very similar and I get the exact same errors in the same place.
    elif self.quantization == "static":
        pass

    self.trainer.save_model('model')

Error A (quantization disabled for Embedding layers):
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/username/project/filename.py", line 402, in predict
    batch_pred_ids, label_ids, _ = self.trainer.predict(dataset)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/trainer.py", line 1355, in predict
    return self.prediction_loop(test_dataloader, description="Prediction")
  File "/home/username/.local/lib/python3.6/site-packages/transformers/trainer.py", line 1417, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only)
  File "/home/username/project/filename.py", line 79, in quant_prediction_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/modeling_bert.py", line 1539, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/modeling_bert.py", line 838, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/modeling_bert.py", line 202, in forward
    embeddings = self.LayerNorm(embeddings)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/quantized/modules/normalization.py", line 25, in forward
    eps=self.eps, output_scale=self.scale, output_zero_point=self.zero_point)
RuntimeError: Could not run 'quantized::layer_norm' with arguments from the 'CPU' backend. 'quantized::layer_norm' is only available for these backends: [QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched,
 VmapMode].

Error B (modified qconfig for embedding layers):
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/username/project/filename.py", line 401, in predict
    batch_pred_ids, label_ids, _ = self.trainer.predict(dataset)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/trainer.py", line 1355, in predict
    return self.prediction_loop(test_dataloader, description="Prediction")
  File "/home/username/.local/lib/python3.6/site-packages/transformers/trainer.py", line 1417, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only)
  File "/home/username/project/filename.py", line 79, in quant_prediction_step
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/modeling_bert.py", line 1539, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/modeling_bert.py", line 838, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/transformers/modeling_bert.py", line 201, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (1024) must match the size of tensor b (128) at non-singleton dimension 0

Errors A seems understandable to me, but I’m not sure what is going on with that second one. If I check manually, without quantization, inputs_embeds, position_embeddings and token_type_embeddings have shape (8, 128, 768), or sometimes (1, 128, 768) at the point where the error occurs. But with QAT (or static quantization), they have shape (1024, 768) or (128, 768) instead, as if the first two dimensions had been concatenated. Is there a way to change them back to their correct shape, maybe by changing the qconfig? I’d rather not edit the library, especially considering that my model type is determined at runtime.

Context:
pie3636:

transformers/modeling_bert.py

This is surprising, can you provide a smaller repro so that we can investigate this further, something like this snippet alone:
   if inputs_embeds is None:
        inputs_embeds = self.word_embeddings(input_ids)
    token_type_embeddings = self.token_type_embeddings(token_type_ids)

    embeddings = inputs_embeds + token_type_embeddings


Question: Hi, I have trained a YOLOv5 QAT model and exported it to ONNX. I noticed in ONNX that the biases are all set to 0. Is this supposed to be the case for QAT modules? Is this due to fuse_modules that causes the biases to be 0 in ONNX?

Context: Hi @MrOCW, afaik there is no supported way to train models using pytorch quantization and then export the quantized models to ONNX. We currently do not plan to add support for ONNX export of pytorch quantized models. If this is important for your use case, feel free to submit a feature request or PR and our team can take a look.


Multi-CPU doesn’t give a speed-up with non-conv parts

Question:
Hi,
I’m trying to quantize torchcrepe module (GitHub - maxrmorrison/torchcrepe: Pytorch implementation of the CREPE pitch tracker)
I used PyTorch’s static quantization.
For all settings, the module size did reduce to about the quarter of the original size. The execution time, however, didn’t decrease always.
With 8 threads,

the execution time rather increased (23.349s → 26.471s)

With 1 thread,

the execution time did decrease (88.641s → 34.877s)

In order to investigate the root cause for the multi-cpu setting, I found the bottleneck–which was the second conv layer–I quantized that part only in a standalone module.
The results were:

the execution time reduced by about 8x for both multi- and single-CPU setting.

Seems like non-conv parts are not helping for speed-up in multi-CPU settings. Would anyone know about this phenomenon? Could you suggest any directions?
Below are the output and the code. Thanks in advance.

Output - Mulit-CPU
CUDA_VISIBLE_DEVICES="" python test.py
========================================================================
=== Before Quantization=================================================
Crepe(
  (conv1): Conv2d(1, 1024, kernel_size=(512, 1), stride=(4, 1))
  (conv1_BN): BatchNorm2d(1024, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv2): Conv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1))
  (conv2_BN): BatchNorm2d(128, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv3): Conv2d(128, 128, kernel_size=(64, 1), stride=(1, 1))
  (conv3_BN): BatchNorm2d(128, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv4): Conv2d(128, 128, kernel_size=(64, 1), stride=(1, 1))
  (conv4_BN): BatchNorm2d(128, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv5): Conv2d(128, 256, kernel_size=(64, 1), stride=(1, 1))
  (conv5_BN): BatchNorm2d(256, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv6): Conv2d(256, 512, kernel_size=(64, 1), stride=(1, 1))
  (conv6_BN): BatchNorm2d(512, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (classifier): Linear(in_features=2048, out_features=360, bias=True)
  (quant): QuantStub()
  (dequant): DeQuantStub()
)
Size (MB):  88.990
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  ../c10/core/TensorImpl.h:1153.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Execution Time (s): 23.349
/opt/conda/lib/python3.8/site-packages/torch/quantization/observer.py:122: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:448.)
  return torch.floor_divide(self, other)
=== After Quantization==================================================
Crepe(
  (conv1): QuantizedConv2d(1, 1024, kernel_size=(512, 1), stride=(4, 1), scale=0.027386803179979324, zero_point=65)
  (conv1_BN): QuantizedBatchNorm2d(1024, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv2): QuantizedConv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.009443704970180988, zero_point=66)
  (conv2_BN): QuantizedBatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv3): QuantizedConv2d(128, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.003322516568005085, zero_point=64)
  (conv3_BN): QuantizedBatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv4): QuantizedConv2d(128, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.001186757697723806, zero_point=66)
  (conv4_BN): QuantizedBatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv5): QuantizedConv2d(128, 256, kernel_size=(64, 1), stride=(1, 1), scale=0.0003566498344298452, zero_point=66)
  (conv5_BN): QuantizedBatchNorm2d(256, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv6): QuantizedConv2d(256, 512, kernel_size=(64, 1), stride=(1, 1), scale=0.0001743721222737804, zero_point=64)
  (conv6_BN): QuantizedBatchNorm2d(512, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (classifier): QuantizedLinear(in_features=2048, out_features=360, scale=0.00038746290374547243, zero_point=62, qscheme=torch.per_channel_affine)
  (quant): Quantize(scale=tensor([0.0079]), zero_point=tensor([0]), dtype=torch.quint8)
  (dequant): DeQuantize()
)
Size (MB):  22.340
Execution time (s): 26.471
==================================================
========================================================================
=== Before Quantization=================================================
ConvModel(
  (conv): Conv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1))
  (quant): QuantStub()
  (dequant): DeQuantStub()
)
Size (MB):  33.556
Execution Time (s): 7.850
=== After Quantization==================================================
ConvModel(
  (conv): QuantizedConv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.022107142955064774, zero_point=56)
  (quant): Quantize(scale=tensor([0.0079]), zero_point=tensor([0]), dtype=torch.quint8)
  (dequant): DeQuantize()
)
Size (MB):  8.394
Execution time (s): 0.983
==================================================

Output - Single-CPU
➜ CUDA_VISIBLE_DEVICES="" python test.py
========================================================================
=== Before Quantization=================================================
Crepe(
  (conv1): Conv2d(1, 1024, kernel_size=(512, 1), stride=(4, 1))
  (conv1_BN): BatchNorm2d(1024, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv2): Conv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1))
  (conv2_BN): BatchNorm2d(128, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv3): Conv2d(128, 128, kernel_size=(64, 1), stride=(1, 1))
  (conv3_BN): BatchNorm2d(128, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv4): Conv2d(128, 128, kernel_size=(64, 1), stride=(1, 1))
  (conv4_BN): BatchNorm2d(128, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv5): Conv2d(128, 256, kernel_size=(64, 1), stride=(1, 1))
  (conv5_BN): BatchNorm2d(256, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (conv6): Conv2d(256, 512, kernel_size=(64, 1), stride=(1, 1))
  (conv6_BN): BatchNorm2d(512, eps=0.001, momentum=0.0, affine=True, track_running_stats=False)
  (classifier): Linear(in_features=2048, out_features=360, bias=True)
  (quant): QuantStub()
  (dequant): DeQuantStub()
)
Size (MB):  88.990
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  ../c10/core/TensorImpl.h:1153.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Execution Time (s): 88.641
/opt/conda/lib/python3.8/site-packages/torch/quantization/observer.py:122: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
  warnings.warn(
/opt/conda/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:448.)
  return torch.floor_divide(self, other)
=== After Quantization==================================================
Crepe(
  (conv1): QuantizedConv2d(1, 1024, kernel_size=(512, 1), stride=(4, 1), scale=0.025484636425971985, zero_point=66)
  (conv1_BN): QuantizedBatchNorm2d(1024, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv2): QuantizedConv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.009398533962666988, zero_point=71)
  (conv2_BN): QuantizedBatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv3): QuantizedConv2d(128, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.003245722968131304, zero_point=58)
  (conv3_BN): QuantizedBatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv4): QuantizedConv2d(128, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.001026029814966023, zero_point=57)
  (conv4_BN): QuantizedBatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv5): QuantizedConv2d(128, 256, kernel_size=(64, 1), stride=(1, 1), scale=0.0004001582565251738, zero_point=72)
  (conv5_BN): QuantizedBatchNorm2d(256, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (conv6): QuantizedConv2d(256, 512, kernel_size=(64, 1), stride=(1, 1), scale=0.00016336666885763407, zero_point=62)
  (conv6_BN): QuantizedBatchNorm2d(512, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  (classifier): QuantizedLinear(in_features=2048, out_features=360, scale=0.00038912895251996815, zero_point=61, qscheme=torch.per_channel_affine)
  (quant): Quantize(scale=tensor([0.0079]), zero_point=tensor([0]), dtype=torch.quint8)
  (dequant): DeQuantize()
)
Size (MB):  22.340
Execution time (s): 34.877
==================================================
========================================================================
=== Before Quantization=================================================
ConvModel(
  (conv): Conv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1))
  (quant): QuantStub()
  (dequant): DeQuantStub()
)
Size (MB):  33.556
Execution Time (s): 31.405
=== After Quantization==================================================
ConvModel(
  (conv): QuantizedConv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1), scale=0.021491192281246185, zero_point=65)
  (quant): Quantize(scale=tensor([0.0079]), zero_point=tensor([0]), dtype=torch.quint8)
  (dequant): DeQuantize()
)
Size (MB):  8.394
Execution time (s): 3.334
==================================================

Code (borrowed from the crepe repo)
import torch
import torchcrepe
import functools
import torch.nn.functional as F

import numpy as np
import os
from torch.quantization import QuantStub, DeQuantStub
import time

class Crepe(torch.nn.Module):
    """Crepe model definition"""

    def __init__(self, model='full'):
        super().__init__()

        # Model-specific layer parameters
        if model == 'full':
            in_channels = [1, 1024, 128, 128, 128, 256]
            out_channels = [1024, 128, 128, 128, 256, 512]
            self.in_features = 2048
        elif model == 'tiny':
            in_channels = [1, 128, 16, 16, 16, 32]
            out_channels = [128, 16, 16, 16, 32, 64]
            self.in_features = 256
        else:
            raise ValueError(f'Model {model} is not supported')

        # Shared layer parameters
        kernel_sizes = [(512, 1)] + 5 * [(64, 1)]
        strides = [(4, 1)] + 5 * [(1, 1)]

        # Overload with eps and momentum conversion given by MMdnn
        batch_norm_fn = functools.partial(torch.nn.BatchNorm2d,
                                          eps=0.001,
                                          momentum=0.0,)

        # Layer definitions
        self.conv1 = torch.nn.Conv2d(
            in_channels=in_channels[0],
            out_channels=out_channels[0],
            kernel_size=kernel_sizes[0],
            stride=strides[0])
        self.conv1_BN = batch_norm_fn(
            num_features=out_channels[0])

        self.conv2 = torch.nn.Conv2d(
            in_channels=in_channels[1],
            out_channels=out_channels[1],
            kernel_size=kernel_sizes[1],
            stride=strides[1])
        self.conv2_BN = batch_norm_fn(
            num_features=out_channels[1])

        self.conv3 = torch.nn.Conv2d(
            in_channels=in_channels[2],
            out_channels=out_channels[2],
            kernel_size=kernel_sizes[2],
            stride=strides[2])
        self.conv3_BN = batch_norm_fn(
            num_features=out_channels[2])

        self.conv4 = torch.nn.Conv2d(
            in_channels=in_channels[3],
            out_channels=out_channels[3],
            kernel_size=kernel_sizes[3],
            stride=strides[3])
        self.conv4_BN = batch_norm_fn(
            num_features=out_channels[3])

        self.conv5 = torch.nn.Conv2d(
            in_channels=in_channels[4],
            out_channels=out_channels[4],
            kernel_size=kernel_sizes[4],
            stride=strides[4])
        self.conv5_BN = batch_norm_fn(
            num_features=out_channels[4])

        self.conv6 = torch.nn.Conv2d(
            in_channels=in_channels[5],
            out_channels=out_channels[5],
            kernel_size=kernel_sizes[5],
            stride=strides[5])
        self.conv6_BN = batch_norm_fn(
            num_features=out_channels[5])

        self.classifier = torch.nn.Linear(
            in_features=self.in_features,
            out_features=torchcrepe.PITCH_BINS)

        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, x, embed=False):

        x = self.quant(x)
        # Forward pass through first five layers
        x = self.embed(x)

        if embed:
            return x

        # Forward pass through layer six
        x = self.layer(x, self.conv6, self.conv6_BN)

        # shape=(batch, self.in_features)
        x = x.permute(0, 2, 1, 3).reshape(-1, self.in_features)

        # Compute logits
        x = torch.sigmoid(self.classifier(x))
        x = self.dequant(x)
        return x

    ###########################################################################
    # Forward pass utilities
    ###########################################################################

    def embed(self, x):
        """Map input audio to pitch embedding"""
        # shape=(batch, 1, 1024, 1)
        x = x[:, None, :, None]

        # Forward pass through first five layers
        x = self.layer(x, self.conv1, self.conv1_BN, (0, 0, 254, 254))
        x = self.layer(x, self.conv2, self.conv2_BN)
        x = self.layer(x, self.conv3, self.conv3_BN)
        x = self.layer(x, self.conv4, self.conv4_BN)
        x = self.layer(x, self.conv5, self.conv5_BN)

        return x

    def layer(self, x, conv, batch_norm, padding=(0, 0, 31, 32)):
        """Forward pass through one layer"""
        x = F.pad(x, padding)
        x = conv(x)
        x = F.relu(x)
        x = batch_norm(x)
        return F.max_pool2d(x, (2, 1), (2, 1)).contiguous()

def print_size_of_model(model):
    torch.save(model.state_dict(), "temp.p")
    print(f'Size (MB): {os.path.getsize("temp.p")/1e6 : .3f}')
    os.remove('temp.p')

class ConvModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(1024, 128, kernel_size=(64, 1), stride=(1, 1))
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.dequant(x)
        return x

torch.set_num_threads(1)

def run(model, input):
    print("="  * (23 + 49))
    print("=== Before Quantization" + "=" * 49)

    # Original
    print(model)
    print_size_of_model(model)
    start = time.time()
    model(input)
    print(f"Execution Time (s): {time.time() - start :.3f}")

    # Quantized
    model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
    torch.quantization.prepare(model, inplace=True)
    # Calibration
    model(input)
    torch.quantization.convert(model, inplace=True)

    print("=== After Quantization" + "=" * 50)
    print(model)
    print_size_of_model(model)
    start = time.time()
    model(input)
    print(f"Execution time (s): {time.time() - start :.3f}")

    print("=" * (23 + 49))

crepe_input = torch.rand(1299, 1024).contiguous()
crepe = Crepe()

conv_model_input = torch.rand([1299, 1024, 128, 1])
conv_model = ConvModel()

# Set to eval()
crepe.eval()
conv_model.eval()

# Disable running stats for measurement
crepe.conv1_BN.track_running_stats = False
crepe.conv2_BN.track_running_stats = False
crepe.conv3_BN.track_running_stats = False
crepe.conv4_BN.track_running_stats = False
crepe.conv5_BN.track_running_stats = False
crepe.conv6_BN.track_running_stats = False

run(crepe, crepe_input)
run(conv_model, conv_model_input)

Context:
Hi @jasonhuh,
This may have something to do with the shape of the input used in the Conv layers. My guess is that if the shape is not big enough then the work per thread is not significant enough for us to see the wins.
@dskhudia do you have any additional insights here about how FBGEMM may be behaving differently for single vs multi-thread here?


Wrong gradients in quantization-aware training

Question: Hi, I use _fake_quantize_learnable_per_tensor_affine api as a component of my quantization layer. I found in some cases, this api will make the gradients of scale, zero_point and input wrong. The gradients generated by backward is different from the ones calculated by hand, even generated nan error. After I replaced  _fake_quantize_learnable_per_tensor_affine with my own written code, the nan disappeared. My pytorch version 1.8.2.

Context: Hi, is there any help?


Question:
Hello,
I am trying to deploy my CNN inpainting model to Android mobile app, so I am following this tutorial.
https://pytorch.org/tutorials/recipes/quantization.html
model = PartialConvUNet()
backend = 'qnnpack'

model.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)
print_size_of_model(model_static_quantized)

And I run this, I get this error.
Traceback (most recent call last):
  File "C:\PycharmProjects\MyCNN\mymodel.py", line 219, in <module>
    torch.backends.quantized.engine = 'qnnpack'
  File "C:\Anaconda3\envs\deeplearning\lib\site-packages\torch\backends\quantized\__init__.py", line 29, in __set__
    torch._C._set_qengine(_get_qengine_id(val))
RuntimeError: quantized engine QNNPACK is not supported

If I set
backend='fbgemm'
It works, but this is the backend for the x86 server, which will not be compatible to Android environment, correct?

Context:
According to this article: A developer-friendly guide to model quantization with PyTorch 4
I needa ARM CPU…

“Since these libraries are architecture-dependent, static quantization must be performed on a machine with the same architecture as your deployment target. If you are using FBGEMM, you must perform the calibration pass on an x86 CPU (usually not a problem); if you are using QNNPACK, calibration needs to happen on an ARM CPU (this is quite a bit harder).”


Which machine should I do the INT8 quantization for mobile deployment?

Question:
Hello,
I recently wrote a custom image inpainting model, an altered form of UNet. Now I want to demo run this model on Android environment, which will be Samsung Galaxy S10. After a few days of research, I got to know that I need to quantize the model for speedy mobile performance. I chose to follow ‘Post training static quantization’ in the link: Quantization Recipe — PyTorch Tutorials 1.10.0+cu102 documentation. But I just encountered a roadblock with a question that, which machine should I run this code below? I wrote this model in x64 Windows 10. Should I run this in pycharm IDE and save the model and then import the model in Android studio? Can someone give me a little bit of guidance to me?
backend = "qnnpack"
model.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)

Development Environment:
OS: Windows 10 x64
CPU: Intel I9-10885H 2.40GHz
GPU: NVIDIA GeForce 1650Ti
PyTorch Version: 1.10.0.dev20210629
Target Environment:
Android OS: 4.4+
Device: Samsung Galaxy S10

Context: Have you taken a look at this page: Android | PyTorch 2?


Training Pre-trained Quantized Resnet 18 model

Question: Hello there. I am new to quantization topic and have been trying to explore this. I have a question about training a pre-trained, quantized Resnet-18 model. Is this possible? If yes, is there a sample on how to do this? Thanks in advance.

Context:
To train this model, you would likely need to convert it back to floating point and then train (because the quantized operators typically do not have autograd support).
I am unaware of an automated way to do this.
Best regards
Thomas


Question:
Hello,
I recently wrote a UNet-like model which receives 2 inputs(input, mask) of same shape and returns 1 output. I am trying to fuse convolutions/batchnorm/Relu like below but it returns error. Is there any way to fuse the input/mask conv layers at the same time?
  for m in model.modules():
        if type(m) == PartialConvLayer:
            torch.quantization.fuse_modules(m, ["input_conv", "mask_conv", "activation"], inplace=True)
            print(type(m))

AssertionError: did not find fuser method for: (<class 'torch.nn.modules.conv.Conv2d'>, <class 'torch.nn.modules.conv.Conv2d'>, <class 'torch.nn.modules.activation.ReLU'>)

The architecture is as below:
PartialConvUNet(
  (encoder_1): PartialConvLayer(
    (input_conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (mask_conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (activation): ReLU()
  )
  (encoder_2): PartialConvLayer(
    (input_conv): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
    (mask_conv): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
    (batch_normalization): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): ReLU()
  )
  (encoder_3): PartialConvLayer(
    (input_conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): ReLU()
  )
  (encoder_4): PartialConvLayer(
    (input_conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): ReLU()
  )
  (encoder_5): PartialConvLayer(
    (input_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): ReLU()
  )
  (encoder_6): PartialConvLayer(
    (input_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): ReLU()
  )
  (encoder_7): PartialConvLayer(
    (input_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): ReLU()
  )
  (decoder_5): PartialConvLayer(
    (input_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_6): PartialConvLayer(
    (input_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_7): PartialConvLayer(
    (input_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_4): PartialConvLayer(
    (input_conv): Conv2d(768, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(768, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_3): PartialConvLayer(
    (input_conv): Conv2d(384, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(384, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_2): PartialConvLayer(
    (input_conv): Conv2d(192, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (mask_conv): Conv2d(192, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_1): PartialConvLayer(
    (input_conv): Conv2d(67, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (mask_conv): Conv2d(67, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  )
)

Context:
Hello!
Think I figured it out. Looks like I did correctly fused some operations?
    for name, module in model.named_modules():
        if type(module) == PartialConvLayer:

            # encoder_1 fusion
            if "encoder_1" in name:
                torch.quantization.fuse_modules(module, [['input_conv', 'activation']], inplace=True)

            # encoder_2 ~ encoder_7 fusion
            elif "enc" in name:
                torch.quantization.fuse_modules(module, [['input_conv', 'batch_normalization', 'activation']], inplace=True)

            # decoder_2 ~ decoder_7 fusion
            elif "decoder_1" not in name:
                torch.quantization.fuse_modules(module, [['input_conv', 'batch_normalization']], inplace=True)

    print("Fusion completed")
    print(model)

This results in,
Fusion completed
PartialConvUNet(
  (encoder_1): PartialConvLayer(
    (input_conv): ConvReLU2d(
      (0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
      (1): ReLU()
    )
    (mask_conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (activation): Identity()
  )
  (encoder_2): PartialConvLayer(
    (input_conv): ConvBnReLU2d(
      (0): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
      (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (mask_conv): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
    (batch_normalization): Identity()
    (activation): Identity()
  )
  (encoder_3): PartialConvLayer(
    (input_conv): ConvBnReLU2d(
      (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (mask_conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): Identity()
  )
  (encoder_4): PartialConvLayer(
    (input_conv): ConvBnReLU2d(
      (0): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (mask_conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): Identity()
  )
  (encoder_5): PartialConvLayer(
    (input_conv): ConvBnReLU2d(
      (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (mask_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): Identity()
  )
  (encoder_6): PartialConvLayer(
    (input_conv): ConvBnReLU2d(
      (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (mask_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): Identity()
  )
  (encoder_7): PartialConvLayer(
    (input_conv): ConvBnReLU2d(
      (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (mask_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): Identity()
  )
  (decoder_7): PartialConvLayer(
    (input_conv): ConvBn2d(
      (0): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (mask_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_6): PartialConvLayer(
    (input_conv): ConvBn2d(
      (0): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (mask_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_5): PartialConvLayer(
    (input_conv): ConvBn2d(
      (0): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (mask_conv): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_4): PartialConvLayer(
    (input_conv): ConvBn2d(
      (0): Conv2d(768, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (mask_conv): Conv2d(768, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_3): PartialConvLayer(
    (input_conv): ConvBn2d(
      (0): Conv2d(384, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (mask_conv): Conv2d(384, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_2): PartialConvLayer(
    (input_conv): ConvBn2d(
      (0): Conv2d(192, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (mask_conv): Conv2d(192, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (batch_normalization): Identity()
    (activation): LeakyReLU(negative_slope=0.2)
  )
  (decoder_1): PartialConvLayer(
    (input_conv): Conv2d(67, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (mask_conv): Conv2d(67, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  )
)


Confused about Torch.Jit.Script error message

Question:
Hello, good morning. I am relatively new to Pytorch but liking it a lot! I have a BERT classification model that I am able to save, load, and run inference on locally, additionally the model is trained using nvidia CUDA GPU. Torch version is 1.10.0+cu102 on Windows 10 64 bit .Up to this point the model loads fine
model.to(device)
model.load_state_dict(torch.load(‘MyModelPath/MyModel15.model’, map_location=torch.device(‘cpu’)))
Here is where the problem starts for me
scripted_model = Torch.Jit.Script(model)
throws this error message
Traceback (most recent call last):
File “C:\Users\javedh\lib\site-packages\IPython\core\interactiveshell.py”, line 3457, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File “”, line 1, in
scripted_model = torch.jit.script(model)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit_script.py”, line 1257, in script
return torch.jit._recursive.create_script_module(
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit_recursive.py”, line 451, in create_script_module
return create_script_module_impl(nn_module, concrete_type, stubs_fn)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit_recursive.py”, line 464, in create_script_module_impl
property_stubs = get_property_stubs(nn_module)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit_recursive.py”, line 778, in get_property_stubs
properties_asts = get_class_properties(module_ty, self_name=“RecursiveScriptModule”)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 161, in get_class_properties
getter = get_jit_def(prop[1].fget, f"__{prop[0]}_getter", self_name=self_name)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 264, in get_jit_def
return build_def(parsed_def.ctx, fn_def, type_line, def_name, self_name=self_name, pdt_arg_types=pdt_arg_types)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 315, in build_def
build_stmts(ctx, body))
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 137, in build_stmts
stmts = [build_stmt(ctx, s) for s in stmts]
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 137, in
stmts = [build_stmt(ctx, s) for s in stmts]
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 287, in call
return method(ctx, node)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 550, in build_Return
return Return(r, None if stmt.value is None else build_expr(ctx, stmt.value))
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 287, in call
return method(ctx, node)
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 702, in build_Call
args = [build_expr(ctx, py_arg) for py_arg in expr.args]
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 702, in
args = [build_expr(ctx, py_arg) for py_arg in expr.args]
File “C:\Users\javedh\AppData\Roaming\Python\Python39\site-packages\torch\jit\frontend.py”, line 286, in call
raise UnsupportedNodeError(ctx, node)
torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren’t supported:
File “C:\Users\javedh\lib\site-packages\transformers\modeling_utils.py”, line 987
activations".
“”"
return any(hasattr(m, “gradient_checkpointing”) and m.gradient_checkpointing for m in self.modules())
I am not sure how to make sense of it in a meaningful way, it seems like a package support issue, but I could be wrong. I would appreciate any assistance in understanding this issue.
Thank you very much!

Context:
The error points to:
torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren’t supported

which is raised due to the used generator expression as seen in this minimal code snippet:
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        a = any(i for i in range(10))
        return x

model = MyModel()
x = torch.randn(1, 1)
out = model(x)

scripted = torch.jit.script(model)
# > UnsupportedNodeError: GeneratorExp aren't supported