# Diverse Pose Lip-Reading Framework
## 5 Experimental Results
### Evaluation Metrics  
The proposed model is trained on the MP dataset and validated on independent test sets. Character error rate (CER) has been used to find error in character recognition, word error rate (WER) to find error in words recognition, and the bilingual evaluation understudy (BLEU) metric is used to find precision in sentences. All these evaluation metrics are used for analysis of the proposed framework. Equation (7) is used to find error rate in characters and words:  
\[\text{ErrorRate }=\frac{(\text{S}+\text{D}+\text{I})}{\text{N}} \tag{7}\]  
where \(S\) in Equation (7) represents the number of substitution errors which occur when a user utters one word, and a recognizer recognizes it as another word. \(D\) is the number of deletion errors which occur when no word is recognized in response to any spoken word. \(I\) is number of insertion errors, which occurs in cases where the word is recognized when none were spoken. \(N\) represents the total number of words in references. The Bilingual Evaluation Understudy (BLEU) is modified n-gram precision that evaluates the closeness of the machine translation to human reference translation. In the case of machine learning based lip reading, it is important to compare the predicted sentences with the ground truth sentences. The BLEU metric can be computed using Equation (8):  
\[pn=\frac{\sum_{\text{C}\in\{\text{Candidates }\}}\sum_{\text{n- gramc}}\text{Count}_{\text{clip}}(\text{n}-\text{ gram })}{\sum_{\text{C}^{\prime}\in\text{(Candidates ) }}\sum_{\text{n- gram }^{\prime}\in\text{C}}\text{Count}\left(\text{n}-\text{ gram }^{\prime}\right)} \tag{8}\]  
The classification accuracy measured by the confusion matrix in Equation (9) is a table that describes the performance of classification model or a classifier for all test sets. In Equation (9), \(TP\) represents \"true positive\", which means that model predicted sentences  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Datasets** & **Types** & **Vocab** & **No. of Uttrances** & **Views** \\ \hline GRID [32] & Phrases & 51 & 33,000 & 0\({}^{\circ}\) \\ \hline LRS [13] & Sentences & 807,375 & 118,116 & 0\({}^{\circ}\)–30\({}^{\circ}\) \\ \hline MV-LRS [13] & Sentences & 14,960 & 74,564 & 0\({}^{\circ}\)–90\({}^{\circ}\) \\ \hline MP dataset & Sentences & 17,500 & 135,442 & 0\({}^{\circ}\) to profile \\ \hline \hline \end{tabular}
\end{table}
Table 2: Vocabulary size with corresponding views in the existing and proposed dataset.  
which are actually spoken and \(TN\) represents a \"true negative\" meaning that a model predicted not any sentence actually when there is silence and just the mouth is open. \(N\) represents total number of sentences:  
\[\text{Accuracy }=\frac{TP+TN}{N} \tag{9}\]  
The videos frames are fed forward into the trained ResNet model and the predictions on lip reading are obtained. The obtained predictions are then compared with the ground-truth, and the values of S and D are computed. These values are fed into Equation (7) to compute CER and WER errors. The predicted lip reading sentences are also evaluated using the BLEU metric to check whether the predicted sentences are similar to the ground-truth.