# Sequential Characteristics Based Operators Disassembly Quantization Method for LSTM Layers
## 4 Experiments
### Comparison with the State of the Art
#### 4.2.2 Word-Level Language Prediction  
In this section, we perform experiments to predict the next word on the Penn Treebank dataset. A one-layer LSTM is verified with d = 300 as in [33], and d = 650 as in [34]. We use the same data preparation and training procedures as in [34]. The optimizer is SGD. The quantization methods are retrained considering the perplexity per word (PPW) metric, which is an index of how \"confused\" the language model is when predicting the next word. For the 2-bit alternating LSTM, only units = 300 are reported in [33]. We proposed a benchmark between our results and other works from the literature. To make the benchmark as fair as possible, we consider that other manuscripts working with LSTM-based models are all based on quantization-aware training (QAT). The focus of our comparison is not on the original floating-point accuracy, but rather on the variation in the metric when applying quantization. Table 4 shows the testing PPW results and size of LSTM parameters with different units. For the 1-bit quantized LSTM, our method and BCN do not achieve comparable performance to the full-precision counterpart. However, in the case of 2-bit quantized LSTM, our method achieves minimum quantization variations of 3.31 and 4.64 compared to the full-precision baseline at the two hidden element counts, respectively, compared to all other methods. Again, in Table 4, we notice that our method leads to the lowest PPW values, 94.81 and 92.24 lower than all comparison methods shown. Notice that the comparison is made in terms of bit widths rather than fixed-point arithmetic because other works do not actually consider the hardware application of the obtained quantized models. Our method, instead, is described considering the subsequent \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{3}{*}{**Precision**} & \multirow{3}{*}{
\begin{tabular}{c} **Quantization** \\ **Method** \\ \end{tabular} } & \multicolumn{3}{c}{**Penn Treebank**} \\ \cline{3-6}  & & & \multicolumn{2}{c}{**Units = 300**} & \multicolumn{2}{c}{**Units = 650**} \\ \cline{3-6}  & & & **PPW** & **Variation** & **PPW** & **Variation** \\ \hline
32-bit & [34] & - & 91.5 & / & 87.6 & / \\  & [26] & BWN & 97.17 & 5.67 & 85.07 & 2.53 \\
1-bit & [25] & BCN & 4330.24 & 4238.74 & 2626.91 & 2539.31 \\  & / & Ours & **157.43** & **65.93** & **135.64** & **48.04** \\ \hline  & [28] & TWN & 95.9 & 4.4 & 92.47 & 4.87 \\
2-bit & / & TCN & 121.79 & 30.29 & 117.35 & 29.75 \\  & [33] & Alternating & 103.1 & 11.6 & - & - \\  & / & Ours & **94.81** & **3.31** & **92.24** & **4.64** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Test PPW and variation of LSTM for word-level language modeling on Penn Treebank. Figure 7: The BPC of our quantization method on different datasets across all varying bits. hardware implementation of our models on architectures completely based on uniform-symmetric quantization strategy. To further assess the effects of the other different quantization methods on the single LSTM Layer, we also make a comparison between the results obtained with the proposed quantization method and the results from other works in the literature by the PPW and relative mean squared error (MSE) metric. In LSTM architectures, we have considered hidden units 300. Table 5 records the PPW and relative MSE of quantized weight matrices with the full-precision. The lowest PPW and relative MSE values are shown in bold. As shown in Table 5, we can see that our proposed method can achieve the lowest metric across all quantization methods. Since quantized models usually have comparable or even better performance than the full-precision baseline, it is worth mentioning that our method is lower than the PPW value of the full-precision model at 4-bit precision. This is by no means uncommon in the contrasting method.