# ClipQ: Clipping Optimization for the Post-Training Quantization of Convolutional Neural Network
## 3 Method
### Weight Clipping Optimization  
The BN layer can accelerate the convergence of the network and alleviate overfitting. It has been widely used in convolutional neural networks [35]. The BN layer also contains floating-point parameters, and they will be absorbed into the convolutional layers before quantization. Specifically, the input \(I\) goes through the convolutional layer and then the BN layer to obtain the output \(O\). \[ \begin{split} O=\gamma\frac{(W*I+b)-\mu)}{\sigma}+\beta\\ =(\gamma\bullet\frac{w}{\sigma})*I+(\gamma\bullet\frac{b-\mu}{ \sigma}+\beta)\end{split}  \tag{2}\]  
Among them, \(*\) represent the convolution and \(\bullet\) represent Hadamard product. \(W\) and \(b\) are the weights and biases of the convolution layer. The mean of the BN layer is \(\mu\). The BN layer's variance is \(\sigma\). \(\gamma\) and \(\beta\) are the learnable parameters of the BN layer. The new weights \(W^{\prime}\) and new bias \(b^{\prime}\) of the merged convolutional layers are as follows:  
\[W^{\prime}=\gamma\bullet\frac{W}{\sigma},b^{\prime}=\gamma\bullet\frac{b-\mu} {\sigma}+\beta \tag{3}\]  
For the convolutional layer, after merging the BN parameters, we quantize its weights channel by channel. That is, each convolution kernel \(w\in R^{i\times kh\times kw}\), where \(i\), \(kh\), and \(kw\) represent the number of channels, height and width of the convolution kernel, and need a group of clipping thresholds. We can obtain the fake-quantized weight \(w_{q}\) according to (1). Most clipping methods determine the clipping thresholds by optimizing the following objective:  
\[ \begin{split}\underset{w_{min},w_{max}}{min}\;distance(w_{q}-w) \end{split}  \tag{4}\]  
where \(distance()\) represents the metric function of quantization deviation. Due to weight decay during network training, the absolute values of weights that have little impact on the loss function will gradually decrease, so the absolute values of weights can be used as a pruning criterion [36; 37; 38]. Inspired by this, we designed dynamic attention for weight clipping, with the optimization objective of the following:  
\[ \begin{split}\underset{w_{min},w_{max}}{min}\;\sum_{l}^{i}\sum_{ m}^{kh}\sum_{m}^{kw}f(w(l,m,n))\\ \times distance(w_{q}(l,m,n)-w(l,m,n))\\ f(x)=\left\{\begin{aligned} &\sqrt{|x|}\; |x|\geq c\\ & min(\sqrt{|w|}),|x|<c\end{aligned}\right.\end{split}  \tag{5}\]  
where \(c\) represents the tertile value of \(|w|\). To put it more clearly, for the weights whose absolute value is ranked in the first \(\frac{1}{3}\), ClipQ gives them the least attention. For the others, the attention varies proportionally with their absolute value. The deviation metric function \(distance()\) adopts L1 distance or L2 distance according to whether it is depthwise  
Figure 1: Linear quantization. convolution. The reason is that the number of parameters of a single channel of depthwise convolution is very small. Equation (5) or Equation (4) is a multi-variable optimization problem. We perform 8-bit linear quantization on a convolution kernel in MobileNetV2 to obtain the numerical relationship between the clipping thresholds and the quantization deviation, as shown in Figure 2. It is a non-convex surface with many local optima, and the global minimum of the surface is expected to be found. Gradient-based optimization methods encounter difficulties in obtaining gradients (due to quantization truncation and rounding) and suffer from the problem of numerous local optima in multivariate optimization. Linear search is a solution (shown in Figure 3a), but it is challenging to determine the appropriate search range and step size, which may result in the global optimum not being within the set interval. Particle swarm optimization, as a swarm intelligence optimization algorithm, can avoid local optima while avoiding problems caused by truncation and rounding in gradient calculations. \[ \begin{split} x_{1}=U(min(w)-\Delta,min(w)+\Delta)\\ x_{2}=U(max(w)-\Delta,max(w)+\Delta)\\ v_{1},v_{2}\sim U(-\frac{L}{20},\frac{L}{20})\\ L=max(w)-min(w)\end{split}