Floating-Point Quantization Process
In integer quantization, the real-valued variable X_{\rm R} is quantized to an integer X_{\rm INT} with the following formula:

 X_{\rm INT}=\alpha\!\left\lfloor{\rm Clip}\!\left(\frac{X_{\rm R}}{\alpha},Q_{min},Q_{max}\!\right)\right\rceil (2)

where \lfloor\cdot\rceil is the rounding function. X_{\rm R} is the real-valued variable, \alpha represents the full-precision scaling factor, and Q_{min} , Q_{max} are the min/max value of the quantization range.
Similarly, a real-valued variable X_{\rm{R}} can be converted to floating-point X_{\rm{FP}} in two steps.

(1) Scale and clip. In FP quantization, we also scale and clip the real-valued variable before quantization as:

\vspace{-0.5em}X_{\rm R}^{\prime}={\rm Clip}\!\left(X_{\rm R},Q_{min},Q_{max}\!\right) (3)

where the min/max value range of signed floating-point quantization can be calculated from Eq. 1 :

Q_{max}=-Q_{min}=(2-2^{-m})2^{2^{e}-b-1} (4)

Here the integer exponent bias b is another adjustable hyperparameter controlling Q_{max} and Q_{min} , which has similar functionality as \alpha . Therefore, for simplicity, we reformulate Eq. 3 as:

\vspace{-0.4em}X_{\rm R}^{\prime\prime}={\rm Clip}\!\left(X_{\rm R},\tilde{Q}_{min},\tilde{Q}_{max}\!\right), (5)

where

\vspace{-0.4em}\tilde{Q}_{max}=\alpha Q_{max} =\alpha\cdot(2-2^{-m})2^{2^{e}-b-1} (6) =\alpha\cdot 2^{-b}\cdot(2-2^{-m})2^{2^{e}-0-1} =2^{-\tilde{b}}\cdot(2-2^{-m})2^{2^{e}-0-1}

Note that we combine the tensor-wise real-valued scaling factor \alpha with integer exponent bias b to form a new scaling factor \tilde{\alpha}=2^{-\tilde{b}}=2^{-b}\cdot\alpha . Here \tilde{b} denotes a relaxed tensor-wise real-valued exponent, and we can derive \tilde{b} from the desired clipping value \tilde{Q}_{max} from Eq. 6 as:

\vspace{-0.4em}\tilde{b}=2^{e}-{\rm log}_{2}{\tilde{Q}_{max}}+{\rm log}_{2}({2-2^{-m}})-1 (7)

(2) Compare and quantize. Different from integer quantization, which simply utilizes the rounding function to convert the real-valued variables to quantized ones, in floating-point quantization, there is an additional step of comparing X_{\rm R}^{\prime\prime} with quantization levels and then quantize:

\vspace{-1em}X_{\rm{FP}}=\tilde{\alpha}\cdot v\cdot\left\lfloor\frac{X_{\rm R}^{\prime\prime}}{\tilde{\alpha}\cdot v}\right\rceil\vspace{-1em} (8)

where X_{\rm R}^{\prime\prime} is clipped real-valued variable (Eq. 5 ), \tilde{\alpha} is the tensor-wise floating-point scaling factor, and v is an integer power of 2.

 v\!=\!\left\{\begin{array}[]{lr}\!\!\!2^{\lfloor\rm{log}_{2}|\mathbf{X}_{\rm R}^{\prime\prime}|+\tilde{b}\rfloor-m}\ \ {\rm if}\ \lfloor\rm{log}_{2}|\mathbf{X}_{\rm R}^{\prime\prime}|\!+\!\tilde{b}\rfloor\!\geq\!1\ \!\!\!2^{1-m}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ {\rm otherwise}\end{array}\right

Here we select the quantization level v according to the magnitude of \frac{X_{\rm R}^{\prime\prime}}{\tilde{\alpha}} , which equals to X_{\rm R}^{\prime\prime}\cdot 2^{\tilde{b}} . Then the floating-point quantized variables can be derived with Eq. 8 . The illustration of the quantization process is in Fig. 1 , detailed explanation can also be found in Micikevicius et al. ( 2022 ) .