아래 유도에서 s*는 최적의 스케일링 백터를 말하는것이 맞는가?

3.2 Protecting Salient Weights by Activation-aware Scaling
We propose an alternative method to reduce the quantization error of the salient weight by per-channel scaling, which does not suffer from the hardware inefficiency issue.
Analyzing the quantization error.
We start by analyzing the error from weight-only quantization. Consider a group/block of weight w; the linear operation can be written as y = wx, and the quantized counterpart is y = Q(w)x. Specifically, the quantization function is defined as:
Q(\mathbf{w})=\Delta\cdot\mathrm{Round}(\frac{\mathbf{w}}{\Delta}),\quad\Delta=\frac{\max(|\mathbf{w}|)}{2^{N-1}},\quad(1)
where N is the number of quantization bits, and Δ is the quantization scaler determined by the absolute maximum value. Now consider a weight element w ∈ w, if we multiply w with s > 1 and the inversely scale x, we will have Q(w · s)(x/s), which is:
Q(w\cdot s)\cdot\frac{x}{s}=\Delta^{'}\cdot\text{Round}(\frac{ws}{\Delta'})\cdot x\cdot\frac{1}{s},\quad(2)
where Δ′ is the new quantization scaler after applying s. We empirically find that: (1) The expected error from Round(·) (denoted as RoundErr(·)) does not change: since the round function maps a floating-point number to an integer, the error is roughly uniformly distributed from [0,0.5], resulting in an average error of 0.25; i.e., RoundErr(·) ∼ 0.25. (2) Scaling up a single element w usually does not change the maximum value from the groupw. Therefore we have Δ′ ≈ Δ; (3) As Δ and x are represented in FP16, they have no quantization error. Consequently, the quantization error from equation 1 and 2 can be expressed as
\begin{aligned}\operatorname{Err}(Q(w)x)&=\Delta\cdot\operatorname{RoundErr}(\frac{w}{\Delta})\cdot x\\\operatorname{Err}(Q(w\cdot s)(\frac{x}{s}))&=\Delta'\cdot\operatorname{RoundErr}(\frac{ws}{\Delta'})\cdot x\cdot\frac{1}{s}\end{aligned}\quad(3)
The ratio of the new error to the original error is \frac{\Delta^{\prime}}\Delta\cdot\frac1s\cdot\text{Given }\Delta^{^{\prime}}\approx\Delta\text{ and }s>1, the relative error is smaller for the salient weight w.
To verify the idea, we multiply the 1% salient channels with s > 1 for the OPT-6.7B model, and measure the change in Δ for each group in Table 2.
We find that scaling up the salient channels is quite effective: the perplexity improves from 23.54 for s = 1 (simply RTN) to 11.92 for s = 2.
As s goes larger, the percentage of changed Δ generally gets larger, but the percentage is still quite small for s < 2 (less than 5%); the relative error for the salient channels continues to go smaller as s increases.
Nonetheless, the best PPL actually appears at s = 2.
This is because if we use a very large s, it will increase the relative error for the nonsalient channels when Δ increases (the error of non-salient channels will be amplified by \frac{\Delta^{\prime}}\Delta, and the ratio is larger than 1 for 21.2% of the channels under s = 4), which can damage the model’s overall accuracy.
Therefore, we need to also consider the error from non-salient channels when protecting salient ones.
Searching to scale. To consider both salient and nonsalient weights, we choose to automatically search for an optimal (per input channel) scaling factor that minimizes the output difference after quantization for a certain layer.
Formally, we want to optimize the following objective:
\mathbf{s^*=\arg\min\mathcal{L}(s)}\\\mathbf{s}\\\mathcal{L}(\mathbf{s})=\|Q(\mathbf{W}\cdot\mathrm{diag}(\mathbf{s}))(\mathrm{diag}(\mathbf{s})^{-1}\cdot\mathbf{X})-\mathbf{W}\mathbf{X}\| (4)
Here Q means the weight quantization function (e.g., INT3/INT4 quantization with group size 128), W is the original weights in FP16, and X is the input features cached from a small calibration set (we take a small calibration set from he pre-training dataset in order not to overfit to a specific task). s is a per-(input) channel scaling factor; for s−1 · X, it can usually be fused into the previous operator (Wei et al., 2022b; Xiao et al., 2022). Since the quantization function is not differentiable, we are not able to directly optimize the problem with vanilla backpropagation. There are some techniques relying on approximated gradients (Bengio et al., 2013; Esser et al., 2019), which we found still suffers from unstable convergence. To make the process more stable, we define a search space for the optimal scale by analyzing the factors that will affect the choice of scaling factor. As shown in the last section, the saliency of weight channels is actually determined by the activation scale (thus “activation-awareness”). Therefore, we simply use a very simple search space:
\mathbf{s}=\mathbf{s_X}^\alpha,\quad\alpha^*=\underset{\alpha}{\operatorname*{\arg\min}}\mathcal{L}(\mathbf{s_X}^\alpha) (5)
s_{X} is the average magnitude of activation (per-channel), and we use a single hyper-parameter α to balance between the protection of salient and non-salient channels. We can find the best α by a fast grid search over the interval of [0, 1] (0 means we do not scale; 1 corresponds to the most aggressive scaling in our search space). We further apply weight clipping to minimize the MSE error of quantization. We provide an ablation study on OPT models under INT3-g128 quantization in Table 5; AWQ consistently outperforms round-to-nearest quantization (RTN) and achieves comparable performance as mixed-precision (1% FP16) while being more hardware-friendly. Advantages. Our method does not rely on any regression (Frantar et al., 2022) or backpropagation, which is required by many quantization-aware training methods. It has minimal reliance on the calibration set since we only measure the average magnitude per channel, thus preventing over-fitting (Figure 8). Therefore, our method requires fewer data for the quantization process and can preserve LLMs’ knowledge outside of the calibration set’s distribution. See Section 5.3 for more details.