Confused about FUSE_RELU flag

WANGSSSSSSS: <p>when i read the source code of quantization operation, I find there exits an outprocess called requantize behind every possible operation,  I guess the purpose of this outprocess is making the activation value bewteen 0-255(maybe my viewpoint is wrong), so the source code uses FUSE_RELU to indicate to update the min vlaue clamp to the zero point , the max value must not bigger than 255.<br>
this is <a href="https://github.com/pytorch/FBGEMM/blob/main/src/FbgemmI8DepthwiseAvx2-inl.h" rel="noopener nofollow ugc">FBgemm outprocess source code</a><br>
Finally, my question is : is this FUSE_RELU flag redundant when the zero point and scale of activation is calculated from relu instead of  the previous conv layer?</p>

HDCharles: <p>the requantization is because the fbgemm kernels for quantized ops accumulate to an int32 value and the output is supposed to be qint8 so it needs to be requantized to that dtype.</p>
<p>I’m not sure what the FUSE_RELU flag does, i think you’d need to ask the FBGEMM team about it.</p>
<p>However, the quantization apis use module fusion in order to get around the fact that for example, if you have a conv followed by a relu, it makes no sense to requantize values that are below 0 since they will be set to 0 anyway, you can get better fidelity by letting the quantization scheme only map from 0 to max_val_of_conv rather than min_val_of_conv to max_val_of_conv like would normally happen. See the fuse_modules step in <a href="https://pytorch.org/docs/stable/quantization.html#static-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.11.0 documentation</a></p>

WANGSSSSSSS: <p>Thanks for reply.<br>
I find the pytorch quantization code convert  convrelu fused module into nniq.ConvRelu2d, which use ops.quantized.conv_relu to forward , so i want to find out the difference between ops.quantized.conv_relu and ops.quantized.conv.<br>
In fact i think the ops.quantized.conv_relu is redundant as the scale and zero point is collected from relu layer.</p>

HDCharles: <aside class="quote no-group" data-username="WANGSSSSSSS" data-post="3" data-topic="153518">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wangsssssss/48/50338_2.png" class="avatar"> WANGSSSSSSS:</div>
<blockquote>
<p>I find the pytorch quantization code convert convrelu fused module into nniq.ConvRelu2d, which use ops.quantized.conv_relu to forward , so i want to find out the difference between ops.quantized.conv_relu and ops.quantized.conv.</p>
</blockquote>
</aside>
<p>They both go to the same place, it just sets or unsets the flag:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/a56f4e23b959fda8e61b9f7c478a300b0b551053/aten/src/ATen/native/quantized/cpu/qconv.cpp#L1426">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/a56f4e23b959fda8e61b9f7c478a300b0b551053/aten/src/ATen/native/quantized/cpu/qconv.cpp#L1426" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/a56f4e23b959fda8e61b9f7c478a300b0b551053/aten/src/ATen/native/quantized/cpu/qconv.cpp#L1426" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/a56f4e23b959fda8e61b9f7c478a300b0b551053/aten/src/ATen/native/quantized/cpu/qconv.cpp#L1426</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="1416" style="counter-reset: li-counter 1415 ;">
          <li>          + c10::to_string(kSpatialDim) + "d, " +</li>
          <li>          "have been removed, please update your model to remove these arguments.");</li>
          <li>      return packed_weight-&gt;apply(act, output_scale, output_zero_point);</li>
          <li>    }</li>
          <li>  }</li>
          <li>};</li>
          <li></li>
          <li>TORCH_LIBRARY_IMPL(quantized, QuantizedCPU, m) {</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv1d"),          QConv1dInt8&lt;false&gt;::run);</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv1d_relu"),     QConv1dInt8&lt;true&gt;::run);</li>
          <li class="selected">  m.impl(TORCH_SELECTIVE_NAME("quantized::conv2d.new"),      QConvInt8&lt;2, false&gt;::run);</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv2d_relu.new"), QConvInt8&lt;2, true&gt;::run);</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv3d.new"),      QConvInt8&lt;3, false&gt;::run);</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv3d_relu.new"), QConvInt8&lt;3, true&gt;::run);</li>
          <li>  // for backward compatibility</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv2d"), QConvInt8ForBC&lt;2, false&gt;::run);</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv2d_relu"), QConvInt8ForBC&lt;2, true&gt;::run);</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv3d"), QConvInt8ForBC&lt;3, false&gt;::run);</li>
          <li>  m.impl(TORCH_SELECTIVE_NAME("quantized::conv3d_relu"), QConvInt8ForBC&lt;3, true&gt;::run);</li>
          <li></li>
          <li>  // transpose</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>you are right though, mathematically it does seem to be redundant, theoretically conv-relu being fused to a convrelu is redundant if the qparams are set so that the range is 0-&gt;max_range</p>

HDCharles: <p><a class="mention" href="/u/kimishpatel">@kimishpatel</a> is there any reason to not do things like <a class="mention" href="/u/wangsssssss">@WANGSSSSSSS</a> suggests, i.e. for any fused op like e.g. Conv3dReLU, since the requant with correct scale and zp would effectively perform the relu for free wouldn’t calling torch.ops.quantized.conv3d be the same/faster than calling torch.ops.quantized.conv3d_relu ?</p>