How do I save and load quantization model

username: Tiru_B
cooked: <p>I have quantized resenet50, quntize_per_channel_resent50 model is giving good accuracy same as floating-point.  If I do torch jit save then I can load torch jit load. and do the inference.</p>
<p>How can I use a torch.save and torch.load model on a quantized model?<br>
Will the entire state dict have same scale and zero points?<br>
How can I get each layer scale and zero points from the quantized model?</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="Tiru_B" data-post="1" data-topic="64188">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tiru_b/48/18731_2.png" class="avatar"> Tiru_B:</div>
<blockquote>
<p>How can I use a torch.save and torch.load model on a quantized model?</p>
</blockquote>
</aside>
<blockquote>
<p>How can I use a torch.save and torch.load model on a quantized model?</p>
</blockquote>
<p>Currently we only support torch.save(model.state_dict()) and model.load_state_dict(…) I think. torch.save/torch.load model directly is not yet supported I believe.</p>
<blockquote>
<p>Will the entire state dict have same scale and zero points?</p>
</blockquote>
<p>No,  they’ll have scale/zero_point that’s calculated from the calibration step.</p>
<blockquote>
<p>How can I get each layer scale and zero points from the quantized model?</p>
</blockquote>
<p>you can print the quantized model and it will show scale and zero_point, e.g.:</p>
<pre><code class="lang-auto">&gt; print(torch.nn.quantized.Conv2d(3, 3, 3))
QuantizedConv2d(3, 3, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0)
</code></pre>

username: Tiru_B
cooked: <aside class="quote no-group" data-username="jerryzh168" data-post="2" data-topic="64188">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p>save</p>
</blockquote>
</aside>
<p>Thank you <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>I was able to save with model.state_dict() but not able to lad the model with same model.load_state_dict(). It was giving keyError.</p>
<p>Secondly if I save with  torch.jit.save(torch.jit.script(pcqmodel),“quantization_per_channel_model.pth”)</p>
<p>I am not able to see the Quantization  info after loading the model . Referred in this issue</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/28331">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/28331" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/28331" target="_blank" rel="noopener nofollow ugc">How to save quantized model in PyTorch1.3 with quantization information</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-10-19" data-time="07:55:01" data-timezone="UTC">07:55AM - 19 Oct 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-10-23" data-time="17:05:26" data-timezone="UTC">05:05PM - 23 Oct 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/vippeterhou" target="_blank" rel="noopener nofollow ugc">
          <img alt="vippeterhou" src="https://avatars.githubusercontent.com/u/29670604?v=4" class="onebox-avatar-inline" width="20" height="20">
          vippeterhou
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: quantization
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## ❓ How to save the quantized model in PyTorch1.3 with quantization information<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">
Is there any way to save the quantized model in PyTorch1.3, which keeps the original information remaining? 

I have known that I can save it after tracing it by:
```python
# Save
torch.jit.save(torch.jit.script(self.model_q), "quant_model.pth")
# Load
mq = torch.jit.load("quant_model.pth")
```
Although `mq` has the right result, it, **however**, losts the quantized information, such as module(layer) name, zero point, scale, etc.


cc @jerryzh168 @jianyuh @dzhulgakov @raghuramank100</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: jerryzh168
cooked: <p>are you using the most recent version? could you try again with PyTorch nightly builds?</p>

username: Zafar
cooked: <p>Also, check if it is just the <code>__repr__</code> that is not showing the info or are the quant params really missing – try getting the scale and zero_point directly.</p>

username: wolffadam
cooked: <p>Be sure you do the whole post training preparation process (by running layer fusion, torch.quantization.prepare() and torch.quantization.convert() ) <strong>before</strong> loading the state_dict.</p>

username: praneet195
cooked: <p>Has this been fixed? I’m unable to save and load quantized models even after following all the steps.</p>

username: Vasiliy_Kuznetsov
cooked: <blockquote>
<p>Has this been fixed? I’m unable to save and load quantized models even after following all the steps.</p>
</blockquote>
<p>do you have a reproducible example on a toy model?</p>