# Imaginary Speech Recognition Using a Convolutional Network with Long-Short Memory
## 4 Results
### Complexity and Memory Analysis  
The tendency of an ISR system is to perform with the highest possible accuracy on a portable device with limited resources. There will always be a tension between the performance and the complexity when developing an ISR system. In this section of the paper, we focused on a study regarding the complexity and memory of the developed system.  
When it comes to the complexity and memory of an intelligent system, the primary resource consumer is generally the neural network. In the case of a CNN neural network with long-term memory, the complexity is given by the long-term memory convolutional layer, due to the need of the implied gates output computation, i.e., the input gate, the forgetting gate, and the output gate, considering the 2D spatial correlation in this process, too. For the 3D convolution corresponding to the input tensor (Nlines \(\times\) Ncolumns \(\times\) Nchannels) and the kernel (klines \(\times\) kcolumns \(\times\) kchannels), the complexity can be measured as:  
\[\begin{array}{c}O((N_{lines}+k_{lines})(N_{columns}+k_{columns})(N_{channels} \\ +k_{channels})\log((N_{lines}+k_{lines})(N_{columns}\\ +k_{columns})(N_{channels}+k_{channels})))\end{array} \tag{16}\]  
when using the fast Fourier transform, as described in [27].  
In our case, the \(N_{lines}\) = \(N_{columns}\), \(k_{lines}\) = \(k_{columns}\), and \(N_{channels}\) = \(k_{channels}\), so the Equation (16) becomes:  
\[O\Big{(}2(N+k)^{2}N_{ch}\log\Big{(}\Big{(}2(N+k)^{2}N_{ch}\Big{)}\Big{)}\Big{)} \tag{17}\]  
where \(N\) is the input lines/columns, \(k\) is the kernel dimension, and \(N_{ch}\) is the number of input channels.  
However, for a ConvLSTM2D layer, this computation (described in Equation (17)), will be made for all the filters of the layer (\(N_{f}\)), for all windows (\(N_{w}\)), and for all gates, plus the computation of the last cell output (a total of four computations). Finally, the complexity can be approximated to:  
\[O\Big{(}4N_{w}\Big{(}2N^{2}N_{ch}\Big{)}\log\Big{(}\Big{(}2N^{2}N_{ch}\Big{)} \Big{)}N_{f}\Big{)} \tag{18}\]  
The details regarding the complexity for all system stages and for all specific layers of the used neural network are presented in Table 10, along with the memory consumption and the execution time measured using an AMD Ryzen 7 4800 HS CPU.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**System Stages**} & **Complexity** & **Memory** & **Time [s]** \\ \hline \multirow{2}{*}{Feature Extraction} & FFT & O(\(N\!/\!f\times N\times\) MlogM) & \textasci{}23 MB & \textasci{}0.001 \\ \cline{2-5}  & COV & O(\(N\!/\!f\times N^{2}\)) & \textasci{}24 MB & \textasci{}0.001 \\ \hline \multirow{5}{*}{CNNLSTM} & ConvLSTM2D-64 & O(\(4\times 4\times(2N^{2})\times\) & \multirow{5}{*}{\(\sim\)2.12 GB} & \multirow{5}{*}{\(0.0819\)} \\  & & log(\(2N^{2}\times 64\)) & & \\ \cline{1-1} \cline{2-5}  & ConvLSTM2D-128 & O(\(4\times 4\times(2N^{2}\times 64)\times\) & \multirow{5}{*}{\(\sim\)2.12 GB} & \multirow{5}{*}{\(0.0819\)} \\  & Dense-64 & O(\(4\times N^{2}\times 128\times 64\)) & & \\ \cline{1-1} \cline{2-5}  & Dense-11 & O(\(64\times 11\)) & & \\ \hline \hline \end{tabular}
\end{table}
Table 10: The complexity and memory of the system detailed for each step taken for system development.