아래 내용에서 3번까지는 이해했는데, 3번에서 4번으로 어떻게 전계가 되는건가?

\begin{aligned}\operatorname{Err}(Q(w)x)&=\Delta\cdot\operatorname{RoundErr}(\frac{w}{\Delta})\cdot x\\\operatorname{Err}(Q(w\cdot s)(\frac{x}{s}))&=\Delta'\cdot\operatorname{RoundErr}(\frac{ws}{\Delta'})\cdot x\cdot\frac{1}{s}\end{aligned}\quad(3)
The ratio of the new error to the original error is \frac{\Delta^{\prime}}\Delta\cdot\frac1s\cdot\text{Given }\Delta^{^{\prime}}\approx\Delta\text{ and }s>1, the relative error is smaller for the salient weight w.
To verify the idea, we multiply the 1% salient channels with s > 1 for the OPT-6.7B model, and measure the change in Δ for each group in Table 2.
We find that scaling up the salient channels is quite effective: the perplexity improves from 23.54 for s = 1 (simply RTN) to 11.92 for s = 2.
As s goes larger, the percentage of changed Δ generally gets larger, but the percentage is still quite small for s < 2 (less than 5%); the relative error for the salient channels continues to go smaller as s increases.
Nonetheless, the best PPL actually appears at s = 2.
This is because if we use a very large s, it will increase the relative error for the nonsalient channels when Δ increases (the error of non-salient channels will be amplified by \frac{\Delta^{\prime}}\Delta, and the ratio is larger than 1 for 21.2% of the channels under s = 4), which can damage the model’s overall accuracy.
Therefore, we need to also consider the error from non-salient channels when protecting salient ones.
Searching to scale. To consider both salient and nonsalient weights, we choose to automatically search for an optimal (per input channel) scaling factor that minimizes the output difference after quantization for a certain layer.
Formally, we want to optimize the following objective:
\mathbf{s^*=\arg\min\mathcal{L}(s)}\\\mathbf{s}\\\mathcal{L}(\mathbf{s})=\|Q(\mathbf{W}\cdot\mathrm{diag}(\mathbf{s}))(\mathrm{diag}(\mathbf{s})^{-1}\cdot\mathbf{X})-\mathbf{W}\mathbf{X}\| (4)
Here Q means the weight quantization function (e.g., INT3/INT4 quantization with group size 128), W is the original weights in FP16, and X is the input features cached from a small calibration set (we take a small calibration set from he pre-training dataset in order not to overfit to a specific task). s is a per-(input) channel scaling factor; for s−1 · X, it can usually be fused into the previous operator (Wei et al., 2022b; Xiao et al., 2022). Since the quantization function is not differentiable, we are not able to directly optimize the problem with vanilla backpropagation. There are some techniques relying on approximated gradients (Bengio et al., 2013; Esser et al., 2019), which we found still suffers from unstable convergence. To make the process more stable, we define a search space for the optimal scale by analyzing the factors that will affect the choice of scaling factor. As shown in the last section, the saliency of weight channels is actually determined by the activation scale (thus “activation-awareness”). Therefore, we simply use a very simple search space:
\mathbf{s}=\mathbf{s_X}^\alpha,\quad\alpha^*=\underset{\alpha}{\operatorname*{\arg\min}}\mathcal{L}(\mathbf{s_X}^\alpha) (5)
s_{X} is the average magnitude of activation (per-channel), and we use a single hyper-parameter α to balance between the protection of salient and non-salient channels. We can find the best α by a fast grid search over the interval of [0, 1] (0 means we do not scale; 1 corresponds to the most aggressive scaling in our search space). We further apply weight clipping to minimize the MSE error of quantization. We provide an ablation study on OPT models under INT3-g128 quantization in Table 5; AWQ consistently outperforms round-to-nearest quantization (RTN) and achieves comparable performance as mixed-precision (1% FP16) while being more hardware-friendly. Advantages. Our method does not rely on any regression (Frantar et al., 2022) or backpropagation, which is required by many quantization-aware training methods. It has minimal reliance on the calibration set since we only measure the average magnitude per channel, thus preventing over-fitting (Figure 8). Therefore, our method requires fewer data for the quantization process and can preserve LLMs’ knowledge outside of the calibration set’s distribution. See Section 5.3 for more details.