# SNA: Sensitivity-Aware Neural Architecture Adaptation for Uniform Quantization
## 3 Methodology
### Initialization  
A good initialization is helpful for the training process to achieve better performance, while a poor initialization can result in slow convergence and high variance, requiring extra time for training to reach satisfactory performance. To alleviate the problem and speed up the quantization-aware fine-tuning process, we explored four different ways to initialize the neural network parameters effectively: (1) **Halving**, (2) **Zero Padding**, (3) **Averaging**, and (4) **Small Int**. The network remains functionally identical when using either **Halving** or **Zero Padding**, which explains why it requires less training time. We explain each method in detail as follows. **Halving**: Inspired by outlier channel splitting (OCS) [38], we divided the values in the first \([C_{\text{in}}\times r]-C_{\text{in}}\) channels by 2 and allocated the extra channels with the same divided values. This approach creates a functionally identical network and reduces the range of the distribution. We can define a convolution layer as in Equation (1). The input size is \((N,C_{\text{in}},H_{\text{in}},W_{\text{in}})\), and the output size is \((N,C_{\text{out}}\,,H_{\text{out}}\,,W_{\text{out}}\,)\). Here, \(N\) represents the batch size, \(C\) denotes the number of channels, \(H\) and \(W\) represent the height and width in pixels, \(\mathbf{W}_{i,j}\) represents the weights, and \(\ast\) represents either matrix multiplication or 2D convolution. For convolution, the size of the weights is \((C_{\text{out}},C_{\text{in}},K_{0},K_{1})\), and \((K_{0},K_{1})\) is the size of the convolving kernel. We analyzed one batch at a time. In the following discussion, the size of \(\mathbf{x}\) is \((C_{\text{in}},H_{\text{in}},W_{\text{in}})\) and the size of \(\mathbf{y}\) is \((C_{\text{out}}\,,H_{\text{out}}\,,W_{\text{out}}\,)\). After channel splitting, the size of \(\mathbf{W}^{\prime}\) is \(([C_{\text{out}}\times r],[C_{\text{in}}\times r],K_{0},K_{1})\). To match the size of the weights, we split the input \(\mathbf{x}=\{\mathbf{x}_{i}\}_{i=0}^{C_{\text{in}}}\) with the formula below:  
\[\mathbf{x^{\prime}}_{i}= \begin{cases}\mathbf{x}_{i}&1\leq i\leq C_{\text{in}} \\ \mathbf{x}_{i-C_{\text{in}}}&C_{\text{in}}+1\leq i\leq[C_{\text{in}}\times r] \end{cases}  \tag{8}\]To ensure the correctness of forward propagation, the output \(\mathbf{y^{\prime}}_{j}\) must also follow the same splitting rule:  
\[\mathbf{y^{\prime}}_{j}= \begin{cases}\mathbf{y}_{j}&1\leq j\leq C_{\text{out}}\\ \mathbf{y}_{j-C_{\text{out}}}&C_{\text{out}}+1\leq j\leq[C_{\text{out}}\times r ]\end{cases}  \tag{9}\]  
Without loss of generality, we halved the values in the first \(s_{in}\) channels (\(s_{in}=[C_{\text{in}}\times r]-C_{\text{in}}\)) and rewrote the convolution equation:  
\[ \begin{split}\mathbf{y}_{j}&=\sum_{i=1}^{s_{in}} \mathbf{x}_{i}*\mathbf{W}_{i,j}+\sum_{i=s_{in}+1}^{C_{\text{in}}}\mathbf{x}_{i}* \mathbf{W}_{i,j}\\ &=\sum_{i=1}^{s_{in}}(\mathbf{x}_{i}*\frac{\mathbf{W}_{i,j}}{2} \times 2)+\sum_{i=s_{in}+1}^{C_{\text{in}}}\mathbf{x}_{i}*\mathbf{W}_{i,j}\\ &=\sum_{i=1}^{s_{in}}\mathbf{x}_{i}*\frac{\mathbf{W}_{i,j}}{2}+ \sum_{i=s_{in}+1}^{C_{\text{in}}}\mathbf{x}_{i}*\mathbf{W}_{i,j}\\ &+\sum_{i=C_{\text{in}}+1}^{[C_{\text{in}}\times r]}\mathbf{x}_{i -C_{\text{in}}}*\frac{\mathbf{W}_{i-C_{\text{in}}j}}{2}\\ &=\sum_{i=1}^{[C_{\text{in}}\times r]}\mathbf{x}^{\prime}_{i}* \mathbf{W}^{\prime\prime}_{i,j}\end{split}  \tag{10}\]  
We split each of the first \(s_{in}\) input channels into two channels, resulting in \(\mathbf{W}^{\prime\prime\prime}\) with a size of \((C_{\text{out}},[C_{\text{in}}\times r],K_{0},K_{1})\). By Halving the weights of these channels, we preserved the equivalence of \(\mathbf{y}_{j}\) and \(\mathbf{y^{\prime}}_{j}\) when \(1\leq j\leq C_{\text{out}}\). For \(\mathbf{y^{\prime}}_{j}\), where \(C_{\text{out}}+1\leq j\leq[C_{\text{out}}\times r]\), we simply duplicated the first \(s_{out}\) output channels (\(s_{out}=[C_{\text{out}}\times r]-C_{\text{out}}\)) of \(\mathbf{W}^{\prime\prime}\).