Question about QAT to 4bit

username: wang_kevin
cooked: <p>Hello!</p>
<p>I am trying to quantize the model to 4bit. My torch version is 1.7.1<br>
I have changed the quant_min and quant_max in qconfig.py, fake_quantize.py, and observer.py (like below)</p>
<pre><code class="lang-auto">    if backend == 'fbgemm':
        qconfig = QConfig(activation=FakeQuantize.with_args(observer=MovingAverageMinMaxObserver,
                                                            quant_min=0,
                                                            quant_max=15,
                                                            reduce_range=True),
                          weight=default_per_channel_weight_fake_quant)
</code></pre>
<pre><code class="lang-auto">default_per_channel_weight_fake_quant = FakeQuantize.with_args(observer=MovingAveragePerChannelMinMaxObserver,
                                                               quant_min=-8,
                                                               quant_max=7,
                                                               dtype=torch.qint8,
                                                               qscheme=torch.per_channel_symmetric,
                                                               reduce_range=False,
                                                               ch_axis=0)
</code></pre>
<pre><code class="lang-auto">if self.dtype == torch.qint8:
    if self.reduce_range:
        quant_min, quant_max = -4, 3
    else:
        quant_min, quant_max = -8, 7

else:
    if self.reduce_range:
        quant_min, quant_max = 0, 7
    else:
        quant_min, quant_max = 0, 15
</code></pre>
<p>I have checked that the range of weights in fake_quantize is correct (In fake_quantize I quantize the weight to check if it is correct or not)</p>
<p>But I get the max value of weights is 8 when I inference the model. It should be 7.</p>
<pre><code class="lang-auto">for layer in model.modules():
        if isinstance(layer, nn.quantized.Conv2d):
            w, b = layer._weight_bias()
            print(w.int_repr().max())
</code></pre>
<p>This is the result of the screenshot.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86f560fa4341e9a0c958c2a518cc41a06702d431.png" data-download-href="https://discuss.pytorch.org/uploads/default/86f560fa4341e9a0c958c2a518cc41a06702d431" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86f560fa4341e9a0c958c2a518cc41a06702d431_2_690x72.png" alt="image" data-base62-sha1="jfTHzjlhGQGo8ugqPXdz7Ik4OHv" width="690" height="72" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86f560fa4341e9a0c958c2a518cc41a06702d431_2_690x72.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86f560fa4341e9a0c958c2a518cc41a06702d431_2_1035x108.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86f560fa4341e9a0c958c2a518cc41a06702d431_2_1380x144.png 2x" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1826×192 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there anything else that needs to be chnaged?</p>
<p>Thanks!</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/wang_kevin">@wang_kevin</a> , your setup looks right so far.  Do you have a small reproducible code snippet you can share to help us look into it?</p>

username: wang_kevin
cooked: <p>Hi <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a> Thanks for the response!</p>
<p>I write the simple code and test get the same problem just like above.</p>
<p>Here is my simple code</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import os

os.environ["CUDA_DEVICE_OR_DER"] = "PCI_BUS_ID"  #see
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
device = torch.device('cuda')

class kernel_conv(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size):
        super(kernel_conv, self).__init__()
        self.quant = torch.quantization.QuantStub()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size)
        self.relu = nn.ReLU()
        self.dequant = torch.quantization.DeQuantStub()
    
    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.relu(x)
        x = self.dequant(x)

        return x

class kernel_fc(nn.Module):
    def __init__(self, in_features, out_features):
        super(kernel_fc, self).__init__()
        self.quant = torch.quantization.QuantStub()
        self.fc = nn.Linear(in_features, out_features)
        self.relu = nn.ReLU()
        self.dequant = torch.quantization.DeQuantStub()
    
    def forward(self, x):
        x = self.quant(x)
        x = self.fc(x)
        x = self.relu(x)
        x = self.dequant(x)

        return x

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = kernel_conv(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = kernel_conv(6, 16, 5)
        self.fc1 = kernel_fc(16 * 5 * 5, 120)
        self.fc2 = kernel_fc(120, 84)
        self.fc3 = nn.Linear(84, 10)
    
    def forward(self, x):
        x = self.pool(self.conv1(x))
        x = self.pool(self.conv2(x))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)

        return x

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

if __name__ == '__main__':
    print("===&gt; creating model &lt;===")
    net = Net()
    net = net.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

    torch.quantization.fuse_modules(net, [["conv1.conv", "conv1.relu"],
                                            ["conv2.conv", "conv2.relu"],
                                            ["fc1.fc", "fc1.relu"],
                                            ["fc2.fc", "fc2.relu"]], inplace=True)
    
    net.eval()

    net.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')

    torch.quantization.prepare_qat(net, inplace=True)

    net.train()

    for epoch in range(1):
        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            optimizer.zero_grad()

            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
                running_loss = 0.0
    
    print('Finished Training')

    net.to('cpu')

    net.eval()

    for layer in net.modules():
        if isinstance(layer, nn.Conv2d):
            weight = layer.weight
            scale = layer.weight_fake_quant.scale
            zero_point = layer.weight_fake_quant.zero_point
            x = torch.quantize_per_channel(weight, scale, zero_point, 0, torch.qint8)
            print('max: ', x.int_repr().max())
            print('min: ', x.int_repr().min())
            break
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c521cf9eeb39a7d8e82972a592afe9a388065923.png" data-download-href="https://discuss.pytorch.org/uploads/default/c521cf9eeb39a7d8e82972a592afe9a388065923" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c521cf9eeb39a7d8e82972a592afe9a388065923_2_690x159.png" alt="image" data-base62-sha1="s7Uv5A13lM69CCLVdNCMHzPfsaL" width="690" height="159" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c521cf9eeb39a7d8e82972a592afe9a388065923_2_690x159.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c521cf9eeb39a7d8e82972a592afe9a388065923_2_1035x238.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/5/c521cf9eeb39a7d8e82972a592afe9a388065923.png 2x" data-dominant-color="131213"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1147×265 84.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: galahad9527
cooked: <p>hello<br>
Are you solve it.<br>
I encountered the same problem.</p>