# SNA: Sensitivity-Aware Neural Architecture Adaptation for Uniform Quantization
## 4 Experimental Results
### Results
The multiplier \(r_{2}\) is \(\sqrt{\frac{4}{3}}\approx 1.155\). The results showed that using channel splitting on ResNet-50-SANA improved the accuracy by 0.420%. With lower quantization bitwidth \(b\) and smaller multiplier \(r\), it becomes more challenging to maintain high performance with limited model capacity. Given that the performance improvement achieved by using \(b_{2}=3\) and \(r_{2}=1.155\) is comparable to that achieved with \(b_{1}=4\) and \(r_{1}=1.414\), we can conclude that the 3 bit quantization for ResNet-50-SANA is more noticeable than the 4 bit quantization for EfficientNet-B2-SANA, as the former task is more challenging. From the perspective of sensitivity analysis, inspired by HAWQ, we utilized PyHessian to compute the sensitivity of a model based on the top Hessian eigenvalue of the model. Our computations yielded values of 19,090.533 for ResNet and 31,197.035 for EfficientNet-B2, clearly illustrating that ResNet is less sensitive than EfficientNet-B2. Because ResNet-50 is relatively redundant, it is less sensitive to quantization, which allowed us to adapt its architecture and quantize it to a lower bitwidth while still maintaining performance. Another interesting observation is that the accuracy of EfficientNet-B2-SANA still dropped by 3.41% even with channel splitting. Since EfficientNet-B2 is very sensitive, the contribution of channel splitting is not enough to bridge the accuracy gap between 3 bit quantization and 4 bit quantization, thus resulting in undesirable performance. Unlike ResNet-50, sensitive networks require larger multipliers to improve model robustness and increase model capacity. As there is no performance gap between W4A8 and W8A8, we concluded that channel splitting can achieve better accuracy with a bitwidth of four and a multiplier of 1.414 for sensitive networks. For such sensitive models, it is advisable to employ a higher quantization bit, minimizing modifications to both the parameters and architectural frameworks. Furthermore, we investigated the effects of different initialization methods on the performance of uniform quantization with the adapted model architecture. We used four initialization schemes: (1) **Halving**, (2) **Zero Padding**, (3) **Averaging**, and (4) **Small Int**. The effect of using different initialization strategies is shown in Table 3. For example, we can juxtapose the ResNet-50-SANA results, featuring W3A8 precision and a 1.55 multiplier, against those of a uniform W4A8 in Table 1, ensuring that the size of the quantized model matches that of the naive W4A8. As can be seen, for EfficientNet-B2-SANA, the sensitivity issues made it difficult to improve the accuracy using W3A8 and a 1.155 multiplier compared with an original EfficientNet-B2 with W4A8 and a 1 multiplier. Interestingly, we observed that the **Zero Padding** method had the highest accuracy for this configuration, yielding a score of 76.942, reflecting a 0.732 increment when compared to the target naive W4A8. Moreover, the **Halving** initialization and the **Small Int** initialization also achieved high accuracy on EfficientNet-B2-SANA, while the **Averaging** initialization showed promising results on ResNet-50-SANA. To compare our approach with other works, we trained a ResNet-50 model that was 2\(\times\) the size of the original ResNet-50, resulting in a model size of 195.6 MB, with an accuracy \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**Mul**} & \multirow{2}{*}{**Size (MB)**} & \multicolumn{6}{c}{**Accuracy**} & \multirow{2}{*}{**Small Int**} & \multirow{2}{*}{**Target**} \\ \cline{5-8}  & & & & & & & & \\ \hline ResNet-50-SANA & W3A8 & 1.155 & 12.2 & 76.630 & **76.942** & **76.834** & **76.622** & **76.210** \\ \hline EfficientNet-B2-SANA & W3A8 & 1.155 & 4.4 & 74.132 & **74.162** & 73.714 & 74.006 & 77.542 \\ \hline EfficientNet-B2-SANA & W4A8 & 1.414 & 8.8 & 78.120 & 77.974 & 78.058 & **78.226** & 78.006 \\ \hline \hline \end{tabular}
\end{table}