Torch.jit.script does not work on a quantized model

username: relaxtheo
cooked: <p>I am a newbie to quantization, and I am trying quantize a model by following the tutorial (<a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.4.0+cu121 documentation</a>), I encountered an error in calling the torch.jit.script(model) in print_size_of_model function for the quantized model, and got following error:</p>
<p>torch.jit.script(model)<br>
/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_check.py:178: UserWarning: The TorchScript type system doesn’t support instance-level annotations on empty non-base types in <code>__init__</code>. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in <code>torch.jit.Attribute</code>.<br>
warnings.warn(<br>
Traceback (most recent call last):<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/_jit_internal.py”, line 385, in get_type_hint_captures<br>
src = inspect.getsource(fn)<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/inspect.py”, line 1139, in getsource<br>
lines, lnum = getsourcelines(object)<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/inspect.py”, line 1121, in getsourcelines<br>
lines, lnum = findsource(object)<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/inspect.py”, line 958, in findsource<br>
raise OSError(‘could not get source code’)<br>
OSError: could not get source code</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_script.py”, line 1432, in script<br>
return _script_impl(<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_script.py”, line 1146, in _script_impl<br>
return torch.jit._recursive.create_script_module(<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_recursive.py”, line 559, in create_script_module<br>
return create_script_module_impl(nn_module, concrete_type, stubs_fn)<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_recursive.py”, line 636, in create_script_module_impl<br>
create_methods_and_properties_from_stubs(<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_recursive.py”, line 468, in create_methods_and_properties_from_stubs<br>
concrete_type._create_methods_and_properties(<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_recursive.py”, line 1004, in try_compile_fn<br>
return torch.jit.script(fn, _rcb=rcb)<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_script.py”, line 1432, in script<br>
return _script_impl(<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_script.py”, line 1204, in _script_impl<br>
fn = torch._C._jit_script_compile(<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/annotations.py”, line 502, in try_ann_to_type<br>
return torch.jit._script._recursive_compile_class(ann, loc)<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_script.py”, line 1602, in _recursive_compile_class<br>
rcb = _jit_internal.createResolutionCallbackForClassMethods(obj)<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/_jit_internal.py”, line 473, in createResolutionCallbackForClassMethods<br>
captures.update(get_type_hint_captures(fn))<br>
File “/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/_jit_internal.py”, line 387, in get_type_hint_captures<br>
raise OSError(<br>
OSError: Failed to get source for &lt;function TreeSpec.<strong>init</strong> at 0x727a3dd68d30&gt; using inspect.getsource</p>
<p>The function print_size_of_model runs successfully for the float resnet model.</p>
<p>What’s the problem for this code, and any advice on fixing it? Thank you</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="relaxtheo" data-post="1" data-topic="207671">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/5fc32e/48.png" class="avatar"> relaxtheo:</div>
<blockquote>
<p>/home/theo/anaconda3/envs/mgi/lib/python3.10/site-packages/torch/jit/_check.py:178: UserWarning: The TorchScript type system doesn’t support instance-level annotations on empty non-base types in <code>__init__</code>. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in <code>torch.jit.Attribute</code>.</p>
</blockquote>
</aside>
<p>torch.jit.script is kind of deprecated, I don’t think we’ll be fixing this, I’d suggest to try out <code>torch.export</code>, our better export system</p>

username: relaxtheo
cooked: <p>Yes, I am trying the export post training quantization (<a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.4.0+cu121 documentation</a>), jit.script is called in print_size_of_model function, do you have new tutorials for export quantization? Thank you</p>

username: jerryzh168
cooked: <p>torch.jit.script is no longer supported, you could write your own size_of_model function probably, we also have <a href="https://github.com/pytorch/ao/blob/174e630af2be8cd18bc47c5e530765a82e97f45b/torchao/utils.py#L196" class="inline-onebox" rel="noopener nofollow ugc">ao/torchao/utils.py at 174e630af2be8cd18bc47c5e530765a82e97f45b · pytorch/ao · GitHub</a> but it may not work 100% for pt2e flow</p>

username: relaxtheo
cooked: <p>get_model_size_in_bytes returns 0 for the quantized_model.</p>
<p>Any advice on how to learn the model quantization? I want to quantize my own model. Thank you</p>

username: relaxtheo
cooked: <p>The model size is not critical, I have another problem in running the pt2e flow according to the guide (<a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.4.0+cu121 documentation</a>), I get following error in the last evaluate call for the loaded_quantized_model:<br>
Expected input at *args[0].shape[0] to be equal to 30, but got 50</p>
<p>Please refer <a href="https://discuss.pytorch.org/t/pt2e-quantized-model-failed-in-evaluating/207691" class="inline-onebox">Pt2e_quantized_model failed in evaluating</a>, hope you can also help me on this. Thank you</p>

username: jerryzh168
cooked: <p>I have some example checking model size in <a href="https://github.com/pytorch/ao/pull/672/files" class="inline-onebox" rel="noopener nofollow ugc">Use `torch.uint1` to `torch.uint7` for Uintx tensor subclass by jerryzh168 · Pull Request #672 · pytorch/ao · GitHub</a></p>