## 2 Related Works
### Scalar Quantization  
Quantization generally means dividing the data into finite levels by transformations and assigning specific values to each level. Among them, scalar quantization is the most basic form of the quantization methods that maps a data to a specific value in the codebook. This scalar quantization is distinguished by the types of intervals in the codebook--uniform or non-uniform. Uniform quantization is a method using quantization intervals which are the same size within a certain range. For example, there is a method of setting the size of the quantization interval by dividing the minimum and maximum values for specific input data by the quantization level. \[Q_{stqp}=(X_{max}-X_{min})/L \tag{1}\]  
\[Q_{indax}(X)=\lfloor(X-X_{min})/Q_{stqp}\rfloor \tag{2}\]  
\[Q(X)=Q_{index}(X)*Q_{stqp}+\frac{Q_{stqp}}{2}+\;X_{min} \tag{3}\]  
Quantization step-size \(Q_{stcp}\) can be calculated using the desired quantization level \(L\) and the maximum range of weights as in Equation (1). The quantized index \(Q_{index}(X)\) of each weight can be obtained by calculating the number of quantization steps apart from the minimum value \(X_{min}\) of weights as shown in Equation (2). Finally, using Equation (3), the quantized vector \(Q(X)\) can be obtained as the center value of each quantization interval. The actual quantization is done by replacing \(X\) with \(Q(X)\). Uniform quantization allows very fast quantization for a given data, but there is a constraint that the distribution of input data must be uniform. In other words, when the distribution of input data is Gaussian distribution, Laplace distribution, or a special distribution such as a gamma distribution, the relative quantization error by using a uniform quantization method can become a loss of performance in CNN. An alternative to this is non-uniform quantization. In non-uniform quantization the value is shifted in order to reduce the quantization error, and the intervals between quantization values are not uniform. Therefore, more sophisticated quantization with relatively low quantization error compared to uniform quantization can be expected. Representative non-uniform quantization is the Lloyd-Max quantizer and k-means clustering. Both of them try to minimize the mean squared error between the original value and the quantized value. The Lloyd-Max quantizer adjusts the quantization value so that the area of the probability density function is the same for each quantization step. Optimized through this approach, the Lloyd-Max quantizer has a dense quantization step in areas with high probability density and, conversely, a wide quantization step in areas with low probability density. \[ \begin{array}{rl}D=\mathbb{E}\Big{[}(x-Q(x))^{2}\Big{]}&=\int\limits_{-\infty }^{\infty}\mathbb{E}\Big{[}(x-Q(x))^{2}\Big{]}f(x)d(x)\\ &=\sum\limits_{i=1}^{L}\int\limits_{-b_{i-1}}^{b_{i}}\mathbb{E}((x-q_{i})^{2} ]f(x)d(x)\end{array}  \tag{4}\]\[b_{i}=\frac{q_{i}+q_{i+1}}{2} \tag{5}\]  
\[q_{i}=\ \frac{\int_{i_{1}}^{b_{i+1}}xf_{x}(x)dx}{\int_{b_{i}}^{b_{i+1}}f_{x}(x)dx} \tag{6}\]  
The Lloyd-Max quantizer adjusts the values of boundary \(b_{i}\) and quantized value \(q_{i}\) so that the mean squared error between the original weights and the quantized weights are minimized, as shown in Equation (4). The two values in Equations (5) and (6) can be obtained by applying a Lagrange condition to Equation (4), \(\frac{\partial D}{\partial b_{i}}=0\), \(\frac{\partial D}{\partial q_{i}}=0\). Another representative methodology for non-uniform quantization is the k-means clustering. Quantization through k-means clustering is one of the main methodologies for quantizing weights of CNN. The quantization using k-means sets the number of quantization bits to \(\log_{2}k\), and based on the result of clustering, the data of each cluster is mapped to the center value of the cluster. \[\underset{C}{\text{argmin}}\ \sum_{i=1,\ \mu^{i}\in M,c^{i}}^{k}\sum_{w^{i}_{j} \in W,c^{i}}\|\mu^{i}-w^{i}_{j}\|^{2} \tag{7}\]  
Equation (7) is the objective of k-means clustering methodology.