## 2 Background
### Quantization  
We are interested in quantization for the purpose of accelerating our structured sub-networks on mobile or edge devices; thus, we construct our \"quantization\" and \"dequantization\" operators from the standard _uniform affine_ mapping from a high precision real number \(r\) to a low-precision quantized number \(q\) using a scaling factor \(s\) and zero-point \(z\), as given by Equation (1). Although more complex non-uniform and non-linear mappings have been considered [4; 25], their utility and practicality are often circumstantial as they require dequantization before performing computation in the high precision real domain and are thus far less efficient on resource-constrained hardware [26].  
\[r=s\cdot(q-z) \tag{1}\]  
As is standard practice, our quantizer (Equation (2)) and dequantizer (Equation (3)) are parameterized by scaling factor \(s\) and zero-point \(z\). Here, \(s\) is a strictly positive real scaling factor and \(z\) is an integer value that maps to the real zero such that the real zero is exactly representable in the quantized domain. This ensures the numerical fidelity of common operations such as zero-padding [27; 28]. Here, \(\lfloor\cdot\rceil\) denotes the half-way rounding function and \(\text{clip}(x;n,p)=\min(\max(x,n),p)\), where \(n\) and \(p\) are the clipping limits defined by the bitwidth \(b\). For signed integers, \(n=-2^{b-1}\) and \(p=2^{b-1}-1\). For unsigned integers, \(n=0\) and \(p=2^{b}-1\). As is standard practice, we use per-tensor scaling factors on the activations and per-channel scaling factors on only the weights [29].  
\[\text{quantize}(x;s,z) :=\text{clip}(\left\lfloor\frac{x}{s}\right\rceil+z;n,p) \tag{2}\] \[\text{dequantize}(x;s,z) :=s\cdot(x-z) \tag{3}\]  
As reported in previous studies, eliminating zero points in the quantization mapping such that \(z=0\) reduces the computational overhead of cross-terms when executing inference using integer-only arithmetic [29; 30]. This strategy is commonly referred to as _symmetric quantization_, with _asymmetric quantization_ as its alternative. To demonstrate the computational overhead of cross-terms introduced by asymmetric quantization, consider the real values for input activation \(x\), weight \(w\), and output activation \(y\) mapping to quantized values \(q_{x}\), \(q_{w}\), and \(q_{y}\), respectively. The arithmetic for their product, \(y=x\cdot w\), becomes the following:  
\[s_{y}(q_{y}-z_{y})=s_{x}(q_{x}-z_{x})\cdot s_{w}(q_{w}-z_{w}) \tag{4}\]  
where \(s_{x}\), \(s_{w}\), and \(s_{y}\) represent their respective scaling factors and \(z_{x}\), \(z_{w}\), and \(z_{y}\) represent their respective zero points. This simplifies to the following:  
\[q_{y}=\frac{s_{x}s_{w}}{s_{y}}(q_{x}q_{w}-q_{x}z_{w}-q_{w}z_{x}+z_{x}z_{w})+z_{y} \tag{5}\]  
These cross-terms often require non-trivial optimizations to remain efficient; however, by constraining the quantization scheme of all weights in a DNN to be symmetric (i.e., \(z_{w}=0\)), we can mask the overhead of asymmetric quantization on the activations without any additional hardware or software optimizations as the arithmetic simplifies to Equation (6), where \(l\) denotes layer \(l\in\{0,\cdots,L\}\) in a neural network with \(L\) sequential layers.  
\[\left(q_{x}^{(l+1)}-z_{x}^{(l+1)}\right)=\frac{s_{x}^{(l)}s_{w}^{(l)}}{s_{x}^{ (l+1)}}q_{w}^{(l)}\left(q_{x}^{(l)}-z_{x}^{(l)}\right) \tag{6}\]  
Note that, when using a symmetric quantizer on both the inputs and outputs to the neural network (i.e., \(x^{(0)}\) and \(x^{(L)}\), respectively), we can freely use asymmetric quantization on the hidden layers without any overhead caused by the cross terms in Equation (5). While previous works have alluded to this optimization [26], we are not aware of any research that has explicitly exploited it as we do. In our work, we focus on uniform affinequantization, where we apply symmetric, per-channel quantization to the weights and asymmetric, per-tensor quantization to the activations.