# SNA: Sensitivity-Aware Neural Architecture Adaptation for Uniform Quantization
## 3 Methodology  
### Motivation  
Uniform quantization is efficient and simple to deploy on general computation hardware compared with mixed-precision quantization. However, mixed-precision quantization can bridge the accuracy gap more effectively under the same memory constraint by assigning bits and discretizing distributions in a non-uniform manner. One of the key differences between these techniques is that uniform quantization overlooks the importance of keeping sensitive and efficient layers of higher precision to save time and resources. In contrast, mixed-precision quantization allocates more bits to more-sensitive layers and fewer bits to less-sensitive layers, resulting in higher performance. To close the accuracy gap, we can either assign higher bitwidths to sensitive layers, as done in mixed-precision quantization, or enhance their robustness to uniform quantization. Taking inspiration from [37], we attempted to reduce the model's sensitivity by splitting its channels. Our hypothesis is that networks with more channels are less sensitive to quantization. There are four reasons supporting our hypothesis. The first is network redundancy. More channels can imply greater redundancy. When quantization is applied, even if some information is lost in a few channels, others can still retain the necessary information for the network to perform well. The second reason is the number of parameters. Networks with more channels usually have more parameters. Even if quantization results in information loss, the overall capability of the model remains. The third reason is feature capture. More channels in the network can capture more features, complex patterns, and higher-order statistics, which might be less affected by quantization. The forth is the model width. Intuitively, wider layers can capture finer-grained patterns on larger images, making them less sensitive to quantization. Our work performed uniform quantization on the adopted models with extra channels, which is simple and straightforward to deploy on general devices and saves computational resources. We accommodated a neural network for uniform quantization using channel splitting and then applied uniform quantization with lower bits on the expanded larger network to fit within the desired memory constraint. Our approach works automatically on the selected models, optimizing the trade-off between quantization and model sensitivity. The whole process is illustrated in Figure 1.  
Consider a layer that takes a vector \(\mathbf{x}=\left\{\mathbf{x}_{i}\right\}_{i=0}^{C_{\text{in}}}\) as the input and a vector \(\mathbf{y}=\left\{\mathbf{y}_{j}\right\}_{j=0}^{C_{\text{out}}}\) as the output. \[\mathbf{y}_{j}=\sum_{i=1}^{C_{\text{in}}}\mathbf{x}_{i}*\mathbf{W}_{i,j} \tag{1}\]  
In this equation, \(\mathbf{W}_{i,j}\) denotes the weights and \(*\) operator stands for either matrix multiplication or 2D convolution. To resize the layer, we introduce a multiplier \(r\) (\(r\in(1,2]\)):  
\[\mathbf{y}^{\prime}_{j}=\sum_{i=1}^{[C_{\text{in}}\times r]}\mathbf{x}^{ \prime}_{i}*\mathbf{W}^{\prime}_{i\bar{\jmath}} \tag{2}\]  
Here, \(\mathbf{x}^{\prime}=\left\{\mathbf{x}^{\prime}_{i}\right\}_{i=0}^{[C_{\text{ in}}\times r]}\) and \(\mathbf{y}^{\prime}=\left\{\mathbf{y}^{\prime}_{j}\right\}_{j=0}^{[C_{\text{ out}}\times r]}\) are the resized input vector and the resized output vector, respectively. \([]\) represents a rounding operation that rounds the number to the nearest integer and truncates the fractional part. To meet the memory constraint, we expanded the entire network using this formula and then uniformly quantized it into a lower bitwidth than the target, rather than quantizing the original network to the target bitwidth. We can approximately estimate the model size and determine the appropriate multiplier \(r\). The number of parameters in \(\mathbf{W}\) would be \(C_{\text{in}}\times C_{\text{out}}\), and the number of parameters in \(\mathbf{W}^{\prime}\) would be \([C_{\text{in}}\times r]\times[C_{\text{out}}\times r]\).