## 5 EI-RNN Optimization and Analysis
### EI-RNN Compression  
The details of the various RNN algorithmic optimizations for edge-based inference are discussed in Section V of Reference [4]. In this section, we briefly present the compression scheme we adopted for edge-mapping studies. As per Reference [28], we carried out train-compress cycles in 3 segments. Training was carried out in the first segment. In the second segment, compression procedures, i.e., hard thresholding, was carried out with low-rank matrix parameterization. Sparse retraining is carried out in the third segment. This process can be formulated in 4 quarters with higher compress and retrain cycles, and the epochs can be increased for finer tuning. The combined list of hyperparameters that affect both training and compression is given in Table 6 with their approximate sensitivity (last column) while performing training (T)-compression (C) cycles. The choice of regularization methods impacts compression. We chose drop out regularization for our experiments, indicated by sparsity indices of matrices W and U. For the 1 st train-test split method, with four datasets and six architectures of RNN, 24 tables of hyperparameter sets were generated. For the 2 nd method of k-fold cross validation, with four datasets and six RNNs, we had another set of 24 tables of hyperparameters. We present 2 of them from method one in Tables 8 and 9. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Si No.**} & \multicolumn{2}{c}{**RNN Unit**} & \multicolumn{2}{c}{**Inference Model Weights**} & \multicolumn{2}{c}{**Inference Model Weights**} \\ \multicolumn{1}{c}{} & \multicolumn{2}{c}{**and Biases with No Rank**} & \multicolumn{2}{c}{**and Biases with Low-Rank Parameterization**} \\ \hline \hline
1 & Basic RNN & W, U, Bh & W1, W2, U1, U2, Bh & \\ \hline
2 & FastGRNN & W, U, Bg, Bh, zeta, nu & W1, W2, U1, U2, Bg, Bh, zeta, nu & W1, W2, U1, U2, B, alpha, beta & W1, W2, U1, U2, B, alpha, beta & \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Inference model parameters of _Edge-Intelligent_ Recurrent Neural Network (EI-RNN) on a Raspberry Pi edge device. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Compression Hyperparameters**} & \multicolumn{5}{c}{**Evaluation Metrics**} & \multicolumn{5}{c}{**Description**} \\ \hline
**uRank** & **wRank** & **sU** & **sW** &
\begin{tabular}{c} **Model\_SIZE** \\ **(KB)** \\ \end{tabular} & **train\_acc** & **val\_acc** & **test\_acc** & **F1 Score** & \\ \hline - & - & 1 & 1 & 5.20 & 0.99 & 0.97 & 0.98 & 0.98 & Baseline \\ \hline
8 & 8 & 1 & 1 & 4.30 & 0.99 & 0.97 & 0.96 & 0.96 & Low-Rank Parameterization (LRP) \\ \hline - & - & 0.9 & 0.9 & 5.00 & 0.99 & 0.98 & 0.98 & Hard thresholding and sparse retraining (HTSR) \\ \hline
16 & 16 & 0.5 & 0.5 & 7.20 & 0.99 & 0.98 & 0.97 & Combination of both LRP and HTSR \\ \hline
8 & 8 & 0.8 & 0.8 & 4.3 & 0.99 & 0.98 & 0.97 & 0.9732 & Memory efficient \\ \hline
12 & 12 & 0.6 & 0.6 & 5.75 & 0.99 & 0.98 & 0.97 & 0.97 & Performance efficient \\ \hline
5 & 5 & 0.9 & 0.9 & 3.2 & 0.98 & 0.97 & 0.96 & 0.96 & Our choice for performance and memory savings \\ \hline
10 & 10 & 0.5 & 0.5 & 5.02 & 0.97 & 0.97 & 0.96 & 0.96 & Other trials \\ \hline
7 & 7 & 0.9 & 0.9 & 3.93 & 0.98 & 0.97 & 0.95 & 0.95 & Other trials \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Compression techniques and hyperparameter checks for Fast Gated Recurrent Neural Network or FastGRNN (with 16 hidden units, 32 batch size, 300 epochs, Adam optimizer) on Daily and Sports Activity Recognition (DSA19) dataset for performance and memory optimization. uRank and wRank are ranks of the associated weight matrice; SU and SW are their sparsity indices. Here, 0.9 indicates 10% sparse, 90% dense. Text in blue represents the best hyperparameter choices.