# Regularized Urdu Speech Recognition with Semi-Supervised Deep Learning
## 4 Results and Analysis
### Neural Network Architecture Analysis  
The optimization and generalization of the neural network model are analyzed with Sigmoid, Relu, Leaky Relu and Maxout activations in the hidden layer. Maxout activations give the best results; however, due to multiple affine maps for Maxout, the number of parameters to be optimized is huge. Hence, a lot of memory resources are required. Using the initial hidden layer with Relu activations and stacking Maxout as the 2 nd hidden layer reduces resource requirements, as the number of parameters to be optimized decreases due to less inputs to the 2 nd hidden layer compared to 1 st.  
Figure 3 records the optimization of the neural network with the single hidden layer using different activation functions. It can be seen that the neural network with sigmoid activations in the hidden layer converges slower than rectified functions and, among rectified functions, Maxout converges fastest due to its power to adapt with activation shapes, but at the cost of huge resources. Leaky and simple Relu, however, are similar in convergence with less computational resource requirements. Figure 4 shows test error with different activation functions. Again sigmoid shows the worst performance while Maxout is the best. Relu and rectified Relu, however, are almost similar.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Validation Data** & **WER\% (Simple NN)** & **WER\% (SSL)** & **WER Decrease** \\ \hline
10\% & 25.68 & 22.08 & 3.6 \\
20\% & 28.17 & 22.68 & 5.49 \\
30\% & 34.79 & 27.34 & 7.45 \\
40\% & 38.88 & 31.25 & 7.63 \\
50\% & 42.82 & 34.65 & 8.17 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Decrease in WER after incorporating Semi-Supervised Learning (SSL) and regularization techniques.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Train** & **Test** & \multicolumn{5}{c}{**WER \%**} \\ \cline{3-7}
**Speakers** & **Speakers** & **Features** & **Adding LLE** & **LLE + ST** & **Ensemble** & **Ensemble + ST** \\ \hline
5–9 & 0–4, & 36.4 & 34.48 & 33.52 & 32.64 & 30.96 \\
0,6–9 & 1–5, & 40.56 & 37.52 & 37.2 & 35.2 & 32.8 \\
0–1,7–9 & 2–6, & 38.8 & 30.8 & 31.52 & 30.16 & 28.64 \\
1–2,8–9 & 3–7, & 40.08 & 37.44 & 33.12 & 35.52 & 32.8 \\
1–3,9 & 4–8, & 50.8 & 49.6 & 46.08 & 45.6 & 41.84 \\
1–4 & 5–9, & 50.32 & 47.2 & 44.32 & 45.84 & 40.88 \\ \hline  & Mean & 42.82 & 39.5 & 37.62 & 37.49 & 34.65 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Word Error Rate for 50% data as test.  
Figure 5 shows optimization convergence of neural networks with single and two hidden layers using different activation functions over the training data of 9 speakers. The optimization is plotted for neural networks with a single Relu hidden layer, two hidden layers, both Relu, and two hidden layers, first Relu and second Maxout. It can be seen that the neural network with two hidden layers, first Relu and second Maxout, converges fastest. Figure 6 shows test error. While all architectures trained on 90% data tend to perform similar on 10% unseen data after steady state, network with Maxout stacked on Relu hidden layer performs better after less iterations of training.  
Figure 4: Test error vs. iterations for single layer activations.  
Figure 5: Training error vs. iterations for network architectures.  
Figure 3: Training error vs. iterations for single layer activations.