Quantization-aware training for GPT2

username: Jamie_Wang
cooked: <p>Hi,</p>
<p>I’m trying to perform QAT on GPT2 model, but I’m a bit confused about the documentation regarding the QuantStub.</p>
<ol>
<li>
<p>Where should I place the QuantStub and DeQuantStub? Based on my understanding I should place the first QuantStub after the Embedding layer and the DequantStub after the Relu activation layer of the FFN; then subsequently the QuantStub will be after the previous DequantStub, which is before the second linear layer of the FFN of the previous decoder layer. Is that correct?</p>
</li>
<li>
<p>I can only fuse the first linear layer and the Relu activation layer of the FFN in each of the decoder layers. Am I right?</p>
</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c8f1c1abaf179cbec43d790d82b3963eca5438d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1c8f1c1abaf179cbec43d790d82b3963eca5438d" title="Decoder-Only-Architecture-used-by-GPT-2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c8f1c1abaf179cbec43d790d82b3963eca5438d_2_423x500.jpeg" alt="Decoder-Only-Architecture-used-by-GPT-2" data-base62-sha1="44DYMV31mNkcedDnjyVGqsto79P" width="423" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c8f1c1abaf179cbec43d790d82b3963eca5438d_2_423x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c8f1c1abaf179cbec43d790d82b3963eca5438d_2_634x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c8f1c1abaf179cbec43d790d82b3963eca5438d.jpeg 2x" data-dominant-color="E2E3E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Decoder-Only-Architecture-used-by-GPT-2</span><span class="informations">722×852 62.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance!</p>
<p>JM</p>

username: M_nh_Le_Dinh
cooked: <p>I am also having problems using Quantization-Aware with GPT-2, did you find a solution? Can you share it with me? Thank you.</p>

username: jerryzh168
cooked: <ol>
<li>I can talk about how to place quantstub/dequantstub in general</li>
</ol>
<p>QuantStub should be placed at the point when you want to quantize a floating point Tensor to a quantized Tensor, the module following QuantStub is also expected to be quantized (to int8 quantized module)</p>
<p>DeQuantStub should be placed at the point when you want to dequantize a int8 tensor back to a floating point Tensor.</p>
<p>e.g.</p>
<pre><code class="lang-python">class M(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(3, 3, 3)
        self.quant = QuantStub
        self.dequant = DeQuantStub()

    def forward(self, x):
        # original input assumed to be fp32
        x = self.quant(x)
        # after quant, x is quantized to int8 tensor
        x = self.conv(x)
        # we also need to quantize conv module to be a int8 quantized conv module
        # which takes int8 Tensor as input and outputs a int8 quantized Tensor
        x = self.dequant(x)
        # dequant would dequantize a int8 quantized Tensor back to a fp32 Tensor
        return x
</code></pre>
<p>Please post the exact model if you need more specific help on the model.</p>
<ol start="2">
<li>Can you include the actual model implementation? in the meantime, you can find all supported fusions here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/fuser_method_mappings.py#L109" class="inline-onebox" rel="noopener nofollow ugc">pytorch/fuser_method_mappings.py at master · pytorch/pytorch · GitHub</a>
</li>
</ol>

username: Jamie_Wang
cooked: <p>Hi Jerry,</p>
<p>Thanks for the explanation.</p>
<ol>
<li>
<p>In your example, the input is quantized from fp32 to int8 by the QuantStub module, but how about the weights in the layer (linear, or conv for example)? It seems that we don’t need to quantize the weight from your example?</p>
</li>
<li>
<p>How about the output from previous layers? For example, the output from the previous linear or activation layer. I understand after the calculation the result should be fp32, so do we need to put QuantStub in between two layers?</p>
</li>
</ol>

username: jerryzh168
cooked: <ol>
<li>for weights, it’s quantized when we swap the floating point conv to quantized conv, it can be done through attaching a qconfig to the conv module instance</li>
<li>there might still be some misunderstanding, a quantized module takes int8 Tensor as input and outputs a int8 Tensor as well, so if previous linear/activation layer is quantized (meaning we attach a int8 qconfig to that layer), we do not need to put a QuantStub in between the two layers.</li>
</ol>