Broken disable_fake_quant

thyeros: <p>It seems disable_fake_quant and other similar functions not working for the following case (Pytorch 1.7).</p>
<pre><code class="lang-auto">class LeNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(28 * 28, 10)    
        self.relu1 = nn.ReLU(inplace=True)

    def forward(self, x):             
        return self.relu1(self.l1(x.view(x.size(0), -1)))  

fq_weight = torch.quantization.FakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   

fq_activation = torch.quantization.FakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   


model = LeNet()
model.l1.qconfig = torch.quantization.QConfig(activation=fq_activation, weight=fq_weight)     
torch.quantization.prepare_qat(model, inplace=True)
model.l1.apply(torch.quantization.disable_fake_quant)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/dabb51fcf16ca81a43ffd628b62f58989d6b1eea.png" data-download-href="https://discuss.pytorch.org/uploads/default/dabb51fcf16ca81a43ffd628b62f58989d6b1eea" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/dabb51fcf16ca81a43ffd628b62f58989d6b1eea.png" alt="image" data-base62-sha1="vcZq3Vvn4zSSJinDEOXL2ddY9aa" width="690" height="180" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1011×265 21.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The above case shows it works just fine. Now, I make my own FakeQuantize as follows, then it doesn’t work any more.</p>
<pre><code class="lang-auto">class MyFakeQuantize(torch.quantization.FakeQuantize):
    def __init__(self, observer, quant_min, quant_max, n_cluster=0, **observer_kwargs):
        super().__init__(observer, quant_min, quant_max, **observer_kwargs)

fq_weight = MyFakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   

fq_activation = MyFakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   


model2 = LeNet()
model2.l1.qconfig = torch.quantization.QConfig(activation=fq_activation, weight=fq_weight)     
torch.quantization.prepare_qat(model2, inplace=True)
model2.l1.apply(torch.quantization.disable_fake_quant)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dda8e92fc095409560814282c6bf6355a0bcd048.png" data-download-href="https://discuss.pytorch.org/uploads/default/dda8e92fc095409560814282c6bf6355a0bcd048" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dda8e92fc095409560814282c6bf6355a0bcd048.png" alt="image" data-base62-sha1="vCTpto5ttnbLmaDtyrzlc1R5DcY" width="690" height="177" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1009×259 21.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this expected behavior?</p>

Vasiliy_Kuznetsov: <aside class="quote no-group" data-username="thyeros" data-post="1" data-topic="106903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thyeros/48/12827_2.png" class="avatar"> thyeros:</div>
<blockquote>
<pre><code class="lang-auto">class MyFakeQuantize(torch.quantization.FakeQuantize):
    def __init__(self, observer, quant_min, quant_max, n_cluster=0, **observer_kwargs):
        super().__init__(observer, quant_min, quant_max, **observer_kwargs)

fq_weight = MyFakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   

fq_activation = MyFakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   


model2 = LeNet()
model2.l1.qconfig = torch.quantization.QConfig(activation=fq_activation, weight=fq_weight)     
torch.quantization.prepare_qat(model2, inplace=True)
model2.l1.apply(torch.quantization.disable_fake_quant)
</code></pre>
</blockquote>
</aside>
<p>hi <a class="mention" href="/u/thyeros">@thyeros</a>, I cannot reproduce this on master.  Could you check if you reproduce this on v1.7 or master?  Which version did you originally see the issue on?</p>

thyeros: <aside class="quote no-group" data-username="thyeros" data-post="1" data-topic="106903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thyeros/48/12827_2.png" class="avatar"> thyeros:</div>
<blockquote>
<pre><code class="lang-auto">class MyFakeQuantize(torch.quantization.FakeQuantize):
    def __init__(self, observer, quant_min, quant_max, n_cluster=0, **observer_kwargs):
        super().__init__(observer, quant_min, quant_max, **observer_kwargs)

fq_weight = MyFakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   

fq_activation = MyFakeQuantize.with_args(\
    observer=torch.quantization.MovingAverageMinMaxObserver.with_args(),
    quant_min=0, quant_max=255, dtype=torch.quint8)   


model2 = LeNet()
model2.l1.qconfig = torch.quantization.QConfig(activation=fq_activation, weight=fq_weight)     
torch.quantization.prepare_qat(model2, inplace=True)
model2.l1.apply(torch.quantization.disable_fake_quant)
</code></pre>
</blockquote>
</aside>
<p>This was with 1.7.0, plz see the snapshot.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40918b8c8a68a947f6bad90bef65c247b63bf920.png" data-download-href="https://discuss.pytorch.org/uploads/default/40918b8c8a68a947f6bad90bef65c247b63bf920" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40918b8c8a68a947f6bad90bef65c247b63bf920_2_356x500.png" alt="image" data-base62-sha1="9dcnZUYx6MNeVy6ScbVDYPaeezu" width="356" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40918b8c8a68a947f6bad90bef65c247b63bf920_2_356x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40918b8c8a68a947f6bad90bef65c247b63bf920_2_534x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40918b8c8a68a947f6bad90bef65c247b63bf920_2_712x1000.png 2x" data-dominant-color="1F2123"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">755×1059 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/thyeros">@thyeros</a>, unfortunately we cannot repro this on master and this is not a known KP.  Could you try the debug script in <a href="https://gist.github.com/vkuzo/61ac9744858509e175d4ce50258782e4" class="inline-onebox" rel="noopener nofollow ugc">gist:61ac9744858509e175d4ce50258782e4 · GitHub</a> and narrow down which exact part of disable_fake_quant is not working in your environment?  In the debug script, I just copied the disable_fake_quant definition so it’s easy to debug.</p>

thyeros: <p>Hi, Vasiliy:</p>
<p>First of all, thanks for the help in this matter. Really appreciated. The code from you doesn’t work out of the box, as FakeQuantizeBase is not available.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/895d337a6727e497767b047750a177cc146e5ff3.png" data-download-href="https://discuss.pytorch.org/uploads/default/895d337a6727e497767b047750a177cc146e5ff3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/895d337a6727e497767b047750a177cc146e5ff3_2_690x95.png" alt="image" data-base62-sha1="jBb5WtJjytDRYD17Gqhg6CQ27g7" width="690" height="95" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/895d337a6727e497767b047750a177cc146e5ff3_2_690x95.png, https://discuss.pytorch.org/uploads/default/original/3X/8/9/895d337a6727e497767b047750a177cc146e5ff3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/9/895d337a6727e497767b047750a177cc146e5ff3.png 2x" data-dominant-color="562628"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">960×133 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Since I know that FakeQuantize is based on FakeQuantizeBase, it looks strange to me as well, hence I checked out the torch/quantization/fake_quantize.py and found out that the pytorch 1.7.0 in my image doesn’t have FakeQuantizeBase and has very different implementations for disable/enable_quant/observer.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf54646d7a60132f2052b5ddc5c43680a7416bd6.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf54646d7a60132f2052b5ddc5c43680a7416bd6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf54646d7a60132f2052b5ddc5c43680a7416bd6_2_518x500.png" alt="image" data-base62-sha1="riA0p2HMS31zmcK4JFyyYp4Sa9g" width="518" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf54646d7a60132f2052b5ddc5c43680a7416bd6_2_518x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf54646d7a60132f2052b5ddc5c43680a7416bd6_2_777x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf54646d7a60132f2052b5ddc5c43680a7416bd6.png 2x" data-dominant-color="1C1415"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">850×819 86.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50e4da9f4de0086ee783e7b6f1a1c622851ee9b4.png" alt="image" data-base62-sha1="bxCvVS55pecm03Mjv6US3T1xhmQ" width="517" height="241"></p>
<p>Can you tell if this is the right one for pytorch1.7.0??</p>

Vasiliy_Kuznetsov: <p>Thanks for the additional info.  Looks like what you are describing was not supported, and added recently with <a href="https://github.com/pytorch/pytorch/pull/48072" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/48072</a>.  Before that PR, the <code>type(mod) == FakeQuantize</code> check would not pass for custom fake quant classes.  It should pass now.  Could you try updating your PyTorch installation and see if it works with 1.7.1 or master?</p>

Vasiliy_Kuznetsov: <p>Looking at <a href="https://github.com/pytorch/pytorch/releases" class="inline-onebox" rel="noopener nofollow ugc">Releases · pytorch/pytorch · GitHub</a>, it doesn’t look like this PR was in 1.7.1.  So, you could try master.  Or, you could write your own function which disables fake_quants (for example, by copying the code after the PR above) and call that, without having to update your PyTorch installation.</p>

thyeros: <p>Thanks Vasiliy for double confirming the latest PR. I will perhaps live with a custom implementation for now (like yours) wait for the official release. Thanks again!</p>