How to adjust the model to eliminate errors in convert_fx()?

jzdcf: <p>Based on the output of symbolic_traced.graph, I have adjusted the model to support symbol tracking. Now that the output of symbolic_traced.graph no longer reports errors, I need to verify what else is needed to ensure that the model can be quantified. What I mean is, the convert_fx() function reports the following error. How to adjust the model ?</p>
<p>from torch.ao.quantization import (<br>
get_default_qconfig_mapping,<br>
get_default_qat_qconfig_mapping,<br>
QConfigMapping,<br>
)<br>
import torch.ao.quantization.quantize_fx as quantize_fx<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b651ea63d35233f36888af6b0ac524539655903d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b651ea63d35233f36888af6b0ac524539655903d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b651ea63d35233f36888af6b0ac524539655903d_2_480x500.png" alt="image" data-base62-sha1="q0SoYXocF1sxFUxpQKeowe8fb7n" width="480" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b651ea63d35233f36888af6b0ac524539655903d_2_480x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b651ea63d35233f36888af6b0ac524539655903d_2_720x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b651ea63d35233f36888af6b0ac524539655903d_2_960x1000.png 2x" data-dominant-color="F1F0EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1166×1214 127 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jerryzh168: <p>this should be supported I think, can you print the error?</p>

jerryzh168: <aside class="quote no-group" data-username="jzdcf" data-post="1" data-topic="205256">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/3ec8ea/48.png" class="avatar"> jzdcf:</div>
<blockquote>
<p>Based on the output of symbolic_traced.graph, I have adjusted the model to support symbol tracking. Now that the output of symbolic_traced.graph no longer reports errors, I need to verify what else is needed to ensure that the model can be quantified. What I mean is, the convert_fx() function reports the following error. How to adjust the model ?</p>
</blockquote>
</aside>
<p>can you checkout our new quantization mode? <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a></p>

jzdcf: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08a7388cb88d623de38a33d7d1bcb52ca06d1814.png" data-download-href="https://discuss.pytorch.org/uploads/default/08a7388cb88d623de38a33d7d1bcb52ca06d1814" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08a7388cb88d623de38a33d7d1bcb52ca06d1814_2_571x500.png" alt="image" data-base62-sha1="1ey5vFrMGbVCh0ZoxwkbrC2d8QA" width="571" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08a7388cb88d623de38a33d7d1bcb52ca06d1814_2_571x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08a7388cb88d623de38a33d7d1bcb52ca06d1814_2_856x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/8/08a7388cb88d623de38a33d7d1bcb52ca06d1814.png 2x" data-dominant-color="F1EEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">899×786 82.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In addition to the fx mode, I also tried the export mode，it reports errors as the picture above<br>
AssertionError:  expecting kwargs for aten op IR to be empty</p>

jerryzh168: <p>can you print the full error? it’s hard to see what is the problem just from the error message</p>

jzdcf: <p>This’s the full error,and my model contained nn.PixelShuffle,Is this operator supported ？</p>
<p>Traceback (most recent call last):<br>
File “/home/user/ecbsr/ECBSR-main/predict.py”, line 63, in <br>
prepared_model = prepare_pt2e(exported_model, quantizer)<br>
File “/home/user/miniconda3/envs/ecbsr/lib/python3.9/site-packages/torch/ao/quantization/quantize_pt2e.py”, line 109, in prepare_pt2e<br>
model = prepare(model, node_name_to_scope, is_qat=False)<br>
File “/home/user/miniconda3/envs/ecbsr/lib/python3.9/site-packages/torch/ao/quantization/pt2e/prepare.py”, line 475, in prepare<br>
_maybe_insert_input_and_output_observers_for_node(node, model, obs_or_fq_map, is_qat)<br>
File “/home/user/miniconda3/envs/ecbsr/lib/python3.9/site-packages/torch/ao/quantization/pt2e/prepare.py”, line 416, in _maybe_insert_input_and_output_observers_for_node<br>
_maybe_insert_input_observers_for_node(<br>
File “/home/user/miniconda3/envs/ecbsr/lib/python3.9/site-packages/torch/ao/quantization/pt2e/prepare.py”, line 383, in _maybe_insert_input_observers_for_node<br>
assert (<br>
AssertionError:  expecting kwargs for aten op IR to be empty</p>

jerryzh168: <aside class="quote no-group" data-username="jzdcf" data-post="7" data-topic="205256">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/3ec8ea/48.png" class="avatar"> jzdcf:</div>
<blockquote>
<p>_maybe_insert_input_observers_for_node</p>
</blockquote>
</aside>
<p>this is the eror: <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/pt2e/prepare.py#L385" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/quantization/pt2e/prepare.py at main · pytorch/pytorch · GitHub</a></p>
<p>could you modify <code>/home/user/miniconda3/envs/ecbsr/lib/python3.9/site-packages/torch/ao/quantization/pt2e/prepare.py</code> to print what is the op? or just add a breakpoint there, it could help further identify the issue</p>

jzdcf: <pre><code class="lang-auto">    assert (
        node.target == torch.ops.aten.clone.default or
        node.target == torch.ops.aten.zeros_like.default or
        len(node.kwargs) == 0
    ), f" expecting kwargs for aten op IR to be empty | {node.target} | {node.kwargs}"
</code></pre>
<p>↑ I modified the source code and printed out the error message： expecting kwargs for aten op IR to be empty | aten.zeros.default | {‘device’: device(type=‘cpu’), ‘pin_memory’: False}</p>
<pre><code class="lang-auto">        # self.mask = torch.zeros((self.out_planes, 1, 3, 3), dtype=torch.float32)
        self.mask = torch.zeros(self.out_planes, 1, 3, 3)
</code></pre>
<p>↑  I modified the parameters ，but still reported the error</p>
<pre><code class="lang-auto">exported_model = symbolic_trace(model)

graph_module = exported_model.graph
print('before delete kargs:',exported_model.graph)

for node in graph_module.nodes:
    if node.op == 'call_function' and node.target == torch.ops.aten.zeros.default:
        new_args = list(node.args)
        new_args[1:] = []  
        node.args = tuple(new_args)

exported_model.recompile()
print('after delete kargs:',exported_model.graph)

quantizer = XNNPACKQuantizer()
quantizer.set_global(get_symmetric_quantization_config())

prepared_model = prepare_pt2e(exported_model, quantizer)
</code></pre>
<p>↑  I manually deleted the extra parameters in the graph</p>
<pre><code class="lang-auto">before quanti size: 87.1884765625
after  quanti size: 125.1064453125
before quanti predict time:32.16 ms
after  quanti predict time:23.80 ms
</code></pre>
<p>↑ The size of the parameters has increased, with only a slight performance improvement</p>

jerryzh168: <p>we don’t expect to see size reduction after prepare_pt2e, can you use <code>convert_pt2e</code> to get a quantized model? also even at that point we still have to lower the model to some backends before we get speedup, see <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html#lowering-and-performance-evaluation" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>

jzdcf: <p>The following code has already been executed, otherwise the model inference would not have executed properly.My question is why there is little change in memory usage when quantifying. When I use Intel NNCF quantization, they point out that the reason is "NNCF does not support quantization of custom PyTorch modules with weights."Does PyTorch have similar restrictions？</p>
<pre><code class="lang-auto">quantized_model = convert_pt2e(prepared_model)
</code></pre>

jerryzh168: <p>can you see the doc: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html#lowering-and-performance-evaluation" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>
<p>the model you are getting after <code>convert_pt2e</code> is this: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html#convert-the-calibrated-model-to-a-quantized-model" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>
<p>so there won’t be perf or memory benefit at that point, you’ll need to lower this model to a specific hardware in order to get benefit</p>