Pre-Shifted Exponent Bias
In transformer architectures, we observed an intriguing phenomenon of high inter-channel variance. As shown in Fig. 2 , the magnitudes of values within the same channel are close to each other but exhibit significant differences across different channels. This phenomenon is not only observed in language models ( i.e., LLaMA and BERT) but also significant in vision transformer models. Since outlier channels are often orders of magnitude bigger than the rest, they will dominate the quantization precision of the quantized tensor, resulting in less representation capacity for those channels with smaller magnitudes Xiao et al. ( 2022 ) . This makes tensor-wise or token-wise scaling factor insufficient for accurate activations quantization.

However, applying per-channel scaling factors for activations poses challenges to efficient matrix multiplication, because the scaling factor is not a shared constant along the multiplication direction and cannot be extracted as Eq. 12 .
To address this challenge, we introduce pre-shifted exponent bias , which allows us to calculate per-channel scaling factors from activations. These scaling factors are then re-parameterized as the exponent biases of the corresponding weights. This method effectively handles high inter-channel variance while maintaining nearly identical efficiency to per-tensor quantization.

Recalling in Eq. 7 , we extracted the tensor-wise integer exponent bias b and times it with real-valued scaling factor \alpha and becomes a new scaling factor \tilde{\alpha}=2^{-\tilde{b}}=2^{-b}\cdot\alpha . Then, the floating-point quantization formula in Eq. 15 becomes:

\vspace{-0.4em}\!\!X_{\rm FP}\!=\!2^{-\tilde{b}}(-1)^{s}2^{p-0}(1+\!\frac{d_{1}}{2}+\frac{d_{2}}{2^{2}}+...+\frac{d_{m}}{2^{m}}) (15)

We note that after the bias is absorbed in the scaling factor, the original bias term ( b^{ori} ) in the FP formula is always zero. In dealing with the inter-channel variance, we devise an innovative usage of this integer exponent bias: we set it to be a per-channel variant ( \mathbf{b}^{ori}\in\mathbb{Z}^{c} ).

Then the calculation of the channel-wise integer bias vector ( \mathbf{b}^{ori} ) is very straightforward. We first calculate the initial per-channel real-valued scaling factor ( 2^{-\tilde{\mathbf{b}}_{j}} ) from the per-channel maximum values:

\vspace{-0.4em}\!\!\!\tilde{\mathbf{b}}_{j}\!=\!2^{e}\!-\!{\rm log}_{2}({{\rm max}(|\mathbf{X}^{:,j}_{\rm R}|})\!)\!+\!{\rm log}_{2}({2\!-\!2^{-m}})\!-\!1 (16)

Here \mathbf{X}^{:,j}_{\rm R} denotes the j^{th} channel in the activation matrix. Then we separate \tilde{\mathbf{b}} to a tensor-wise real-valued scaling factor plus a channel-wise integer scaling factor:

\vspace{-0.4em}\tilde{\mathbf{b}} =\tilde{\rho}+\mathbf{b}^{ori} (17) =\tilde{\rho}+clip(\lfloor\tilde{\mathbf{b}}-\tilde{\rho}\rceil,0,2^{e-1})

where \tilde{\rho}\in\mathbb{R}^{1} , \mathbf{b}^{ori}\in\mathbb{Z}^{c} .
Then the formula for one of the entries in the j^{th} channel of \mathbf{X} can be rewrote as follows:

\vspace{-0.4em}\!\!X_{\rm FP} \!=\!2^{-\tilde{\mathbf{b}}_{j}}(-1)^{s}2^{p-0}(1+\frac{d_{1}}{2}+...+\frac{d_{m}}{2^{m}}) (18) \!=\!2^{-\tilde{\rho}}(-1)^{s}2^{p-\mathbf{b}^{ori}_{j}}(1+\frac{d_{1}}{2}+...+\frac{d_{m}}{2^{m}})

Note that the bias \mathbf{b}^{ori} is constrained to integers within [ 0,2^{e}-1 ], compatible with the standard floating-point number calculation.
Nevertheless, adding different biases for each channel during inference may still cause some extra hardware operations. Thus, we re-parameterized the per-channel activation bias into a weight tensor and pre-computed the weights using the calibration set. This way, the exponent biases shifting only happens in the calibration stage. Then, an element in j^{th} channel of activation tensors X becomes:

\!\!X_{\rm FP} \!=\!2^{-\!\tilde{\rho}}(-1)^{s}2^{p-0}(1\!+\!\frac{d_{1}}{2}\!+\!...\!+\!\frac{d_{m}}{2^{m}}) (19)

and the corresponding weight element in j^{th} row of the weight tensor W becomes:

\!\!\!W_{\rm FP}\!=\!2^{-\tilde{\mathbf{b}}^{\!W}}\!(-\!1)^{s}2^{p-\mathbf{b}^{ori}_{j}}(1\!+\!\frac{d_{1}}{2}\!+\!...\!+\!\frac{d_{m}}{2^{m}}) (20)

As result, efficient matrix multiplication in Eq. 12 is reformulated as:

\!\!\mathbf{O}_{out}^{i,k}\!=\!\mathbf{X}_{\rm FP}^{i,:}\mathbf{W}_{\rm FP}^{:,k}=\tilde{\alpha}_{{}_{\mathbf{X}}}\tilde{\alpha}_{{}_{\mathbf{W}}}^{k}\mathbf{\tilde{X}}^{i,:}_{\rm FP}(\beta\odot\mathbf{\tilde{W}}^{:,k}_{\rm FP}) (21)

where \odot is the element-wise multiplication, \bm{\beta}=2^{-\mathbf{b}^{ori}} and (\bm{\beta}\odot\mathbf{\tilde{W}}^{:,k}_{\rm FP}) can be pre-calculated and stored in low-bit FP format. We depict the overall pre-shifted exponent bias method in Fig. 3 . This method applies to quantizing all the fully-connected layers.
During the search process, we initialize \tilde{\rho}_{{}_{\mathbf{X}}} as the {\rm min}_{j}(\tilde{\mathbf{b}}_{j}) . Then, we fixed \tilde{\mathbf{b}}_{{}_{\mathbf{X}}} to be the bias calculated from the Eq. 16 and search for the optimal \tilde{\rho}_{{}_{\mathbf{X}}} from [ \gamma_{{}_{1}}\tilde{\rho}^{\ init}_{{}_{\mathbf{X}}},\gamma_{{}_{2}}\tilde{\rho}^{\ init}_{{}_{\mathbf{X}}} ].

Combining pre-shifted exponent bias method with the joint format and max-value search framework( FPQ baseline ), we name our method as ( FPQ ), short for Floating Point Quantization .