# Enrichment of Oesophageal Speech: Voice Conversion with Duration-Matched Synthetic Speech as Target
## 5 Subjective Test  
While unprocessed OS has several undesired artefacts and lacks a natural fundamental frequency, it is natural speech. On the other hand, although the BLSTMSS outputs are much clearer sounding, they are synthetically produced and may have some limitations because of that. The success of the enrichment depends majorly on whether listeners prefer to listen to the enriched version more than the unprocessed OS. Therefore, we performed a preference test to collect listeners' opinion on whether they prefer listening to the outputs of the proposed system or the unprocessed OS. The preference test was a 5-point Comparison Mean Opinion Scores (CMOS) test conducted using a web-based interface ([[https://aholab.ehu.eus/users/sneha/BLSTMSS_evaluation/preference_test.php](https://aholab.ehu.eus/users/sneha/BLSTMSS_evaluation/preference_test.php)]([https://aholab.ehu.eus/users/sneha/BLSTMSS_evaluation/preference_test.php](https://aholab.ehu.eus/users/sneha/BLSTMSS_evaluation/preference_test.php)) (accessed on 25 June 2021)). A web-based test was considered more appropriate owing to COVID restrictions. Participants were sourced by sending emails to speech technology networks in Spain and other local networks. The participants were instructed to perform the test with headphones. They were informed that there are no correct or incorrect answers in the test and that they should state their opinions with full liberty. The participants listened to 10 pairs of sentences from each of the four speakers--a total of 40 pairs of sentences. Each pair contained one unprocessed OS sentence and the corresponding BLSTMSS output of the same sentence. The chosen 10 pairs were the shortest sentences in the set, as that allowed us to have the maximum number of evaluations while keeping the test under 20 min. The presentation order of all the pairs, as well as the order of the BLSTMSS and OS sentences within each pair was randomised to avoid order bias. After listening to the two stimuli in each pair, the participants were asked to mark the preferred stimulus. To do so, they were given the following options: 'Prefiero claramente la primera' (I clearly prefer the first one), 'Prefiero la primera' (I prefer the first one), 'No percibo differencia/Ninguna suena mejor' (I do not perceive any difference/Neither one sounds better), 'Prefiero la segunda' (I prefer the second one), 'Prefiero claramente la segunda' (I clearly prefer the second one). Apart from the 40 test pairs, there were 4 pairs (presented at regular intervals) where both the samples were the same file, which was a sentence spoken by a healthy  
Figure 7: STOI scores for Unprocessed OS, previous systems (PPG and BLSTMHS) and proposed BLSTMSS system for speaker 02M3. Reference signal for STOI is duration–matched SS. Error bars show standard errors. speaker. As both the files in these 4 control pairs were the same exact file, we expected the participants to mark the third option ('I do not perceive any difference/Neither one sounds better'). Only those participants who correctly marked this option for at least 3 of these 4 control pairs, were included in the analysis. This ensured reliability of the participants' responses. We asked the participants to describe the audio equipment they used during the test. This was to ensure that they were not using any bad equipment. The options were: good headphones, normal headphones, good loudspeakers, normal loudspeakers and bad equipment. We also asked whether the participants had any experience with using speech technologies. The options were: no experience, experts, sporadic users and through perception tests. This was not to study the effect of speech expertise on the evaluations, but to ensure a good mix of all kinds of listeners. A total of 32 native Spanish participants performed the listening test. Two of them were rejected because they failed the control test. One other participant described their audio equipment as 'bad equipment' and was excluded too.