# SNA: Sensitivity-Aware Neural Architecture Adaptation for Uniform Quantization
## 3 Methodology
### Initialization
Thus, the extra weights are equal to the corresponding weights of the first \(s_{out}\) output channels. \[ \begin{split}\mathbf{y^{\prime}}_{j}&=\mathbf{y}_{j -C_{\text{out}}}=\sum_{i=1}^{[C_{\text{in}}\times r]}\mathbf{x}^{\prime}_{i}* \mathbf{W}^{\prime\prime}_{i,j-C_{\text{out}}}\\ &=\sum_{i=1}^{[C_{\text{in}}\times r]}\mathbf{x}^{\prime}_{i}* \mathbf{W}^{\prime}_{i,j}\end{split}  \tag{11}\]  
We expanded certain channels in the weights and scaled the layer with factor \(r\) while preserving functional equivalence. To achieve this, we halved and duplicated the values in the first \(s_{in}\) channels (\(s_{in}=[C_{\text{in}}\times r]-C_{\text{in}}\)) to obtain \(\mathbf{W}^{\prime\prime}\). After duplication, we obtain \(\mathbf{W}^{\prime\prime}_{i,j-C_{\text{out}}}=\mathbf{W}^{\prime}_{i,j}\) for \(C_{\text{out}}+1\leq j\leq[C_{\text{out}}\times r]\). The size of \(\mathbf{W}^{\prime}\) is \(([C_{\text{out}}\times r],[C_{\text{in}}\times r],K_{0},K_{1})\). In Figure 2, we illustrate the process of neural architecture adaption with **Halving** initialization. **Zero Padding:** In digital signal processing, Zero Padding is typically employed to make the length of a signal a power of two by extending the signal with zeros. Inspired by this, we initialized the weights of the extra channels to zeros to keep the generality of the model. After splitting inputs \(\mathbf{x}\) and \(\mathbf{y}\) as Equations (8) and (9), we can rewrite the convolution equation as in Equation (10). Note that \(\mathbf{0}_{C_{\text{out}}\times([C_{\text{in}}\times r]-C_{\text{in}})\times K _{0}\times K_{1}}\) is an all-zero matrix with a size of \((C_{\text{out}},[C_{\text{in}}\times r],K_{0},K_{1})\). \[ \begin{split}\mathbf{y}_{j}&=\sum_{i=1}^{C_{\text{in }}}\mathbf{x}_{i}*\mathbf{W}_{i,j}+\sum_{i=C_{\text{in}}+1}^{[C_{\text{in}} \times r]}\mathbf{x}_{i-C_{\text{in}}}*\mathbf{0}_{C_{\text{out}}\times([C_{ \text{in}}\times r]-C_{\text{in}})\times K_{0}\times K_{1}}\\ &=\sum_{i=1}^{[C_{\text{in}}\times r]}\mathbf{x}^{\prime}_{i}* \mathbf{W}^{\prime\prime}_{i,j}\end{split}  \tag{12}\]By initializing extra channels with zeros, we obtain \(\mathbf{W}^{\prime\prime}{}_{i,j}\), whose size is \((C_{\text{out}},[C_{\text{in}}\times r],K_{0},K_{1})\). Then, we duplicated \(\mathbf{W}^{\prime\prime}{}_{i,j}\) (\(1\leq j\leq s_{out}\)) and expanded the output channels as specified in Equation (11). The weights of the scaled convolutional layer are denoted by \(\mathbf{W}^{\prime}{}_{i,j}\). With **Zero Padding**, the layer remains functionally equivalent while extending the original input and output size \(r\)-times. This technique enables the effective usage of the pre-trained weights for the adopted architecture, which requires a larger model size. **Averaging**: In the Averaging initialization, we followed the same splitting rule as described earlier (12). However, here, we cannot guarantee the equivalency. Given the fact that stochastic gradient descent during training incrementally updates the network weights to minimize the loss, one intuitive strategy is to allocate the extra channels with the average value of pre-trained model parameters. Assuming that the weights of the best base model and the optimal expanded model come from the same distribution, Averaging can potentially lead to faster convergence and less fine-tuning time. For \(1\leq i\leq C_{\text{in}}\) and \(1\leq j\leq C_{\text{out}}\), we have \(\mathbf{W}^{\prime}{}_{i,j}=\mathbf{W}_{i,j}\), while for \(C_{\text{out}}+1\leq j^{\prime}\leq[C_{\text{out}}\times r]\) or \(C_{\text{in}}+1\leq i^{\prime}\leq[C_{\text{in}}\times r]\), we initialized the extra channels using the formulas below:  
\[\mathbf{W}^{\prime\prime}{}_{i,j^{\prime}}=\mu_{\text{out}} \tag{13}\]  
where \(\mu_{\text{out}}\) is the mean value of the weights for the output channels. Then, we split the input channels.