## 2 Background and Related Work
### Ugrnn  
Unitary Gated RNNs or UGRNNs provide stable training and address the vanishing and exploding gradient problems by limiting the range of the singular values of the hidden state transition matrix. This structure leads to higher RNN prediction accuracy but has impacted the model size compared to FGRNNs and FRNNs. However, they are simpler than LSTM and GRU, as shown in Figure 6. The internal gating structure follows Equations (20)-(22).  
\[z_{t}=\sigma\left[U_{zh}\ h_{t-1}+W_{zx}\ x_{t}+b_{z}\right] \tag{20}\] \[\bar{h}_{t}=\tanh\left[U_{hh}\ h_{t-1}+W_{hx}\ x_{t}+b_{h}\right]\] (21) \[h_{t}=z_{t}\bigodot h_{t-1}+(1-z_{t})\bigodot\bar{h}_{t} \tag{22}\]  
Porting dense and architecturally complex RNNs on resource-constrained edge devices is not directly feasible. Memory and computation spaces on edge devices are far smaller than those of CPU, GPU, or cloud-based machines. Compressing neural networks is one of the solutions to this problem.  
The existing studies have employed Neural Architecture Search (NAS), pruning [29; 30], and quantization [14] for edge-based DL deployment. TinyNAS and TinyEngine [13] as CNN models' MCUs have been recent and popular advancements. Pruning is a common and efficient strategy used for model compression. Pruning refers to removing redundant connections in a network [31] while retaining the accuracy level of the task defined. This results in sparse [32] and irregular data flow computations from a hardware perspective [33]. Techniques that shrink the network without such side effects have been beneficial [34; 35]. Therefore, the weight matrices become structured (blocked) [36; 37] or unstructured [14] based on the pruning technique.  
Low-Rank Matrix Factorization (LMF) expresses a base matrix A of dimension \(\mathrm{m}\times\mathrm{n}\) as a product of two smaller matrices \(\mathrm{U}\) and \(\mathrm{V}\) of dimensions \(\mathrm{m}\times\mathrm{d}\) and \(\mathrm{d}\times\mathrm{n}\), respectively.  
Figure 5: Internal structure of Fast Recurrent Neural Network Unit.  
Parameter d controls the compression factor [38]. The low-rank parameterization of weight matrices has resulted in smaller models while producing mixed results in maintaining model accuracy. A novel optimization algorithm via low-rank constraint and sparsity projection is discussed in Ref. [39]. The low-rank and diagonalization of weight matrices were adopted in Ref [40]. Ref. [41] studied mechanisms for learning compact RNNs and LSTMs via low-rank factorizations and parameter-sharing schemes. Ref. [42] employed the Singular Value Decomposition (SVD) technique on a recurrent projection matrix of a speech model to map onto an embedded platform. Ref. [41] applied low-rank factorization and projection compression technique on large-scale vocabulary speech signal modeling at a small cost of 0.3% increase in Word Error Rate (WER) but a significant 75% decrease in model parameters.  
Quantization refers to tuning the precision of the weights and activations of a network to lower levels while retaining accuracy. It is a very common step in memory optimization for RNNs. Ref. [29] employed quantization of weights using the k-means clustering mechanism. Ref. [43] compared floating to fixed point behavior of neural network parameters and their impact on speedup for a speech recognition task. The details of various RNN algorithmic optimizations for edge-based inference are discussed in Section V of Ref. [4]. Quantization, pruning, and tensor decomposition methods are some of the common methods used for compressing RNNS and CNNs [44].