# Regularized Urdu Speech Recognition with Semi-Supervised Deep Learning
## 2 Literature Review
### Semi-Supervised Learning  
Besides supervised regularization techniques, SSL has also been proposed for generalization. SSL incorporates unsupervised learning techniques to utilize huge amounts of unlabeled data and uses the obtained information to aid supervised classification tasks [11]. Examples include constrained clustering, classification of input data with reduced dimensions and multitasking hybrid networks. Constrained clustering uses unsupervised clustering of complete data; it groups together closer data points but does so by adding constraints to cluster data points labeled with the same classes in the same groups [12]. Classification of datasets with dimensions reduced by unsupervised dimensionality reduction also improves generalization as it utilizes structures in complete data sets, whether labeled or unlabeled [13]. Multitasking hybrid networks is a proposed technique of sharing classification network layers with an unsupervised auto-encoder network which tries to regenerate unlabeled input data at the output [14]. Self-training is a simple SSL technique based on iterative training using confidently predicted test samples. Most confident test samples predicted by initially trained models are added to training sets along with their predicted labels as targets, and the model is trained for a few more iterations on the enhanced training set [15]. We have utilized self-training of neural network models using data points predicted with high probability using a semi-supervised clustering technique called label propagation. Label propagation is a famous graph-based semi-supervised clustering algorithm and utilizes both labeled and unlabeled data. All data points are represented by fully a connected graph with edges indicating their geometric distance. In the first step, class labels from all labeled points are transferred to connected points with the label transfer probability assigned as inversely proportional to their distance. Net probability for each label against a point after this step is the sum of transfer probabilities of that label from connected points normalized across all labels. The same is repeated for unlabeled data points until convergence of label probabilities [16]. Label transfer probability from point i to point j is computed by Equation (1). k are all labeled and unlabeled points, and graph edges between points i and j have weight Dij indicating the distance between i and j.  
\[\text{Tij}=\text{ P(i }\rightarrow\text{j)}=\frac{\text{Dij}}{\sum_{k}\text{Dki}} \tag{1}\]  
Usually Euclidean distance is used as the distance metric between two points; however, any metric can be used provided it gives positive values, as distance values are used to measure the probability of label transfer. Probability Pc for a point i against each class c is obtained by normalizing the sum of transition probabilities Tc for that class across transition probabilities for all classes. Equations (2) and (3) are used to find label transition probabilities and net class probabilities for each point, respectively. \[\text{Tc}=\text{ }\sum_{\text{j}}\text{Tij} \tag{2}\]  
\[\text{Pc}=\text{Tc}/\text{ }\sum_{c}\text{Tc} \tag{3}\]  
Unsupervised dimensionality reduction is usually used to obtain important features from the data. Training on reduced dimensions improves the generalization of neural networks by utilizing the structures of labeled as well as unlabeled data [17]. Linear dimensionality reduction techniques project data on linear axes. Due to the nonlinear nature of speech, linear techniques are not very useful for ASR. Nonlinear dimensionality reduction, on the other hand, projects data onto a manifold that is a nonlinear, smooth and curved subspace within Euclidean space. Every manifold can be considered as linear very close to a point. LLE uses this and finds weights for a point in order to be represented linearly by k nearest neighbors. These weights contain information about local structures in the neighborhood of points.