# Enrichment of Oesophageal Speech: Voice Conversion with Duration-Matched Synthetic Speech as Target
## 4 Evaluations and Results
### ASR  
We evaluated the outputs of our proposed enrichment system using three ASR systems: the speech-to-text system from Microsoft Azure using the python azure-cognitiveservices-speech library (ASR 1) [42], the Elhuyar speech recognition system (ASR 2) [43] and a Kaldi [36] based system (ASR 3) developed in our laboratory and described in [44]. The input files to these ASR systems were the 100 single channel wav files sampled at 16,000 Hz. The outputs were text files containing the transcriptions. The reason for using three ASR systems was to have a diverse set of evaluations. ASR 1 is a well known commercial ASR system used world wide and therefore easier for comparisons in future studies elsewhere. ASR 2 is a commercial system built locally in Spain, and therefore, better adapted to the speech style and vocabulary of the speakers involved in this study. ASR 3 is a customised Kaldi based ASR with full control of all thecomponents such as the language model, dictionary etc. We presume that the amount of audio used to train ASR 3 (approximately 5 h of audio) was smaller in comparison to the other two commercial systems. It uses a lexicon limited to the vocabulary of the corpus used in this study. It also uses a unigram language model and was used in our previous studies [31; 32]. The advantage of ASR 3 is that it is not prone to updates as is the case of commercial ASRs. This allows us to make fair and accurate comparisons of our ongoing work with our previous work. We calculated two metrics from the ASR transcriptions: Word Error Rates (WER) and Percentage Words Correct (PWC). For WER, we calculated the Levenshtein distance [45] between the reference sentence (original recording utterance text) and the hypothesis sentence (ASR transcription output) using the Word Error Rate Matlab toolbox [46]. The WER formula is shown in Equation (1). The Levenshtein distance WER takes into account the insertions, deletions, and substitutions that are observed in the transcribed output. Please note that the WER can be more than 100% if the total insertions, substitutions and deletions exceed the total number of words in the reference sentence. \[\text{WER}=\frac{\text{Substitutions}+\text{Insertions}+\text{Deletions}}{ \text{Total number of words in reference sentence}}\times 100. \tag{1}\]  
PWC is the percentage of words from the reference sentence correctly identified in the transcribed sentence. The PWC formula is shown in Equation (2). \[\text{PWC}=\frac{\text{Words correctly identified in transcription}}{\text{Total number of words in reference sentence}}\times 100. \tag{2}\]  
Figures 2-4 show mean WER and PWC scores for the 100 sentences obtained from the transcriptions of ASR 1, 2 and 3, respectively. WER scores were lower (i.e., higher intelligibility) for BLSTMSS compared to unprocessed OS for all ASRs and speakers with 2 exceptions--speaker 04M3 in ASR 1 and speaker 16M3 in ASR 2. In the case of PWC scores, a higher PWC score (i.e., higher intelligibility) was observed for the BLSTMSS samples compared to unprocessed OS samples for all speakers and ASRs. When comparing the different ASR systems, the best WER and PWC scores for unprocessed OS were obtained by ASR 1, followed by ASR 3 and ASR 2. In addition, there were fewer differences between OS and enriched OS in ASR 1 compared to the other two systems. Amongst all the ASRs, ASR 3 had the best WER and PWC scores for enriched OS. We did correlation analysis of WERs and PWCs of OS obtained from ASR 1. There was a significant negative correlation ([PERSON]'s \(r=-0.959\), \(p=0.041\)) between WER and the number of months since laryngectomy and a significant positive correlation ([PERSON]'s \(r=0.952,p=0.048\)) between PWC and the number of months since laryngectomy. A similar correlation was observed with speaking rate, but it did not reach significance. In our previous studies [31; 32], we worked with speaker 02M3 and ASR 3. Figure 5 shows the WER scores of BLSTMSS in comparison to our previous methods, PPG [32] and BLSTMHS [31].