Convert floating point 32 bit of input and pretrained weight to 8bit

Ganga: <ul>
<li>
<p>I am using alexnet model ,where 7layers are binarized(input and weight),1st layer is not binarized(input,weight are floating point 32 bit).I want  only 1st layer input,weight to be converted to 8 bit  before sending into the convolution function without harming the other.</p>
</li>
<li>
<p>i am using pretrained weight here</p>
</li>
</ul>

Zafar: <p>Just to make it clear – when you say “convert to 8bit” are you using quantization or are you just casting the types down? Also, we don’t support quantization lower than 8 bits, so binarization of the layers might not be supported without custom hacks.</p>
<p>Lastly, if you already have the weights, and you just need an 8-bit model, you can follow these steps:</p>
<ol>
<li>Make sure your model is quantizable – all layers in your network must be stateful and unique, that is, no “implied” layers in the forward and no inplace computation</li>
<li>Prepare the model using <code>prepare</code> function</li>
<li>Calibrate the prepared model by running through your data AT LEAST once</li>
<li>Convert your model to the quantized version.</li>
</ol>
<p>You can follow the PTQ tutorial here: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a></p>
<p>On the first point:</p>
<p>This model cannot be quantized:</p>
<pre><code class="lang-python">class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.relu = nn.ReLU(inplace=True)
  def forward(self, a, b):
    ra = self.relu(a)
    rb = self.relu(b)
    return ra + rb
</code></pre>
<p>To make the model quantizable, you need to make sure there are no inplace operations, and every operation can save the state:</p>
<pre><code class="lang-python">class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.relu_a = nn.ReLU(inplace=False)
    self.relu_b = nn.ReLU(inplace=False)
    self.F = nn.quantized.FloatFunctional()
  def forward(self, a, b):
    ra = self.relu_a(a)
    rb = self.relu_b(b)
    return self.F.add(ra, rb)
</code></pre>
<p>If you want to have the model take FP input and return the FP output you will need to insert the <code>QuantStub</code>/<code>DequantStub</code> at the appropriate locations:</p>
<pre><code class="lang-python">class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.quant_stub_a = torch.quantization.QuantStub()
    self.quant_stub_b = torch.quantization.QuantStub()
    self.relu_a = nn.ReLU(inplace=False)
    self.relu_b = nn.ReLU(inplace=False)
    self.F = nn.quantized.FloatFunctional()
    self.dequant_stub = torch.quantization.DeQuantStub()
  def forward(self, a, b):
    qa = self.quant_stub_a(a)
    qb = self.quant_stub_b(b)
    ra = self.relu_a(qa)
    rb = self.relu_b(qb)
    return self.dequant_stub(self.F.add(ra, rb))
</code></pre>
<p>Similarly, if you would like to only quantize a single layer, you would need to place the quant/dequant only where you want to quantize. Please, note that you would need to specify the quantization parameters appropriately:</p>
<pre><code class="lang-python">class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.quant_stub_a = torch.quantization.QuantStub()
    self.relu_a = nn.ReLU(inplace=False)
    self.relu_b = nn.ReLU(inplace=False)
  def forward(self, a, b):
    qa = self.quant_stub_a(a)
    ra = self.relu_a(qa)
    a = self.dequant_stub(ra)
    rb = self.relu(b)
    return ra + rb
</code></pre>
<p>The model above will be partially quantizable, and you would need to give the qconfig to the quant_stub and the relu only.</p>