# SNA: Sensitivity-Aware Neural Architecture Adaptation for Uniform Quantization
## 3 Methodology
### Initialization
For \(C_{\text{in}}+1\leq i^{\prime}\leq[C_{\text{in}}\times r]\), we have:  
\[\mathbf{W}^{\prime}{}_{i^{\prime},j}=\mu_{\text{in}} \tag{14}\]  
where \(\mu_{\text{in}}\) is the mean value of the weights for the input channels, whose output dimensions are \([C_{\text{out}}\times r]\). **Small Int**: Using the average value of layer weights may lead to a very large initialization for low-bit quantization. To address this, we divided the average value by the  
Figure 2: Illustration of neural architecture adaptation with **Halving** initialization. The top left picture illustrates the convolution process, while the bottom picture illustrates the convolution process after **Halving** initialization. The top right picture explains why **Halving** initialization maintains functional equivalence. In this picture, we use dotted lines to represent channel splitting and numbers to differentiate between duplicating and Halving. The number 1 denotes duplicating, while the fraction \(\frac{1}{2}\) denotes Halving. real-valued scaling factor \(S\) when performing uniform quantization. Thus, the weights are mapped to small integers to quicken the process during quantization. In addition, here, we used the statistics of the weight distribution instead of the actual weight values for initialization. Similar to **Averaging**, for \(1\leq i\leq C_{\text{in}}\) and \(1\leq j\leq C_{\text{out}}\), we have \(\textbf{W}^{\prime}_{i,j}=\textbf{W}_{i,j}\). For \(C_{\text{out}}+1\leq j^{\prime}\leq\left[C_{\text{out}}\times r\right]\) or \(C_{\text{in}}+1\leq i^{\prime}\leq\left[C_{\text{in}}\times r\right]\), we initialized the extra channels using the formula below:  
\[\textbf{W}^{\prime\prime}_{i,j^{\prime}}=\frac{\max(|\mu_{\text{out}}-3\times \sigma_{\text{out}}|,|\mu_{\text{out}}+3\times\sigma_{\text{out}}|)}{2^{b-1}} \tag{15}\]  
where \(\sigma_{\text{out}}\) is the standard deviation of the weights for the output channels. After channel splitting, for \(C_{\text{in}}+1\leq i^{\prime}\leq\left[C_{\text{in}}\times r\right]\), we initialized \(\textbf{W}^{\prime}_{i,j}\) with:  
\[\textbf{W}^{\prime}_{i,j}=\frac{\max(|\mu_{\text{in}}-3\times\sigma_{\text{in }}|,|\mu_{\text{in}}+3\times\sigma_{\text{in}}|)}{2^{b-1}} \tag{16}\]  
Here, \(\sigma_{\text{in}}\) is the standard deviation of the weights for the input channels, whose output dimensions are \(\left[C_{\text{out}}\times r\right]\). Specifically, in symmetric quantization, the quantized newly initialized weights \(\textbf{W}^{\prime}_{i^{\prime},j}\) are:  
\[ \begin{split} Q(r)&=[\frac{r}{S}]=[\frac{\textbf{W }^{\prime}_{i^{\prime},j}}{\max(|\textbf{W}^{\prime}_{i^{\prime},j}|)}]\\ &\approx[\frac{\max(|\mu_{\text{in}}-3\times\sigma_{\text{in}}|, |\mu_{\text{in}}+3\times\sigma_{\text{in}}|)}{\max(|\textbf{W}^{\prime}_{i^{ \prime},j}|)]}\end{split}  \tag{17}\]  
Assuming that the weights \(\textbf{W}^{\prime}_{i^{\prime},j}\) follow a Gaussian distribution, during inference, the weights of the extra channels are mapped to 1 owing to the three-sigma rule. In most situations, the weights of the extra channels are mapped to small integers such as 0, 1, and 2, which can speed up the quantization process. In summary, we explored four initialization schemes in total: (1) **Halving**, (2) **Zero Padding**, (3) **Averaging**, and (4) **Small Int**. After initialization, we fine-tuned the expanded model to improve its robustness. With these allocation strategies, the expanded model inherits the pre-trained weights, keeps the generality, and requires less hyperparameter tuning to reach a satisfactory performance. As shown in Section 4, our initialization strategies achieved decent results with low bitwidth weights, which shows the superiority of uniform quantization with the adapted model architectures. It should be noted that we can easily implement these strategies by modifying the neural network building blocks, which makes these strategies easy to apply to different networks.