## 9 Experimental Assessment and Analysis
### Experiment 2: Subjective Evaluations  
A measure of the quality of speech cannot be given exclusively through objective measures as peculiar characteristics such as intelligibility, naturalness, fluidity and intonation can only be assessed from a human listener through subjective measures. Naturalness describes how close synthesized speech is to human speech and is usually measured by a mean opinion score (MOS) test [68]. In a MOS test, subjects listen to speech and rate its relative perceived quality on some kind of a scale, for example, \(5=excellent\), \(4=good\), \(3=fair\), \(2=poor\), \(1=bad\). Then the scores are averaged across subjects. However, \"naturalness\" may be difficult to judge in any type of speech. More recent authors, like [PERSON] [69], suggest that _naturalness_ be measured by presenting to a group of listener a couple of sentences, one spoken by humans and the other produced by an algorithm, in our case the _FuzzyGenetic_. It is important to note that human speech is low pass filtered in order to make it quite worse than the original. Each listener must then decide whether the speech is produced by a human or a synthesizer. We used a much simpler criterion, that is, to say whether the naturalness of the heard word is _acceptable_ (POSITIVE) or not (NEGATIVE). Fifty listeners listened to a series of artificial words giving their degree of naturalness, comprehension and synchronization. The generated words include 120 meaningless words and 120 words with meaning. Moreover, all these words are also reproduced by a talking head animated with the facial movements obtained with the \(Fuzzy-Genetic\) algorithm. As a comparison, we considered the DECTalk speech synthesizer [70]. Naturalness data of DECTalk is taken from Reference [71]. Recall that we assume that the visual synthesis is performed with the algorithm described in Reference [60]. Figure 16: Spectrograms of the original (**top**) and artificial (**bottom**) word /\(p\)\(a\)\(t\)\(i\)\(t\)\(a\)/. In the following we give a very short explanation on how the results on audio-visual synthesis are obtained. From Reference [71], the DCTalk subjective measure for audio nonsense words of 47% corresponds, using the data in Reference [58], to an Equivalent Noise Level of about \(-13\) dB. Adding visual information to the synthetic speech the subjective measure increases to 95% but, considering the performance of the visual system, this means an expected average subjective score of 77.08%. Likewise, a subjective score of 57% for words with meaning synthesized with DCTalk leads to an expected improvement to 79.96% with the audio-visual system. We show in the following the results of intelligibility test on the \(Fuzzy-Genetic\) synthetic speech with comparison with another system. Intelligibility of synthetic voices can be measured with one of the many available tests; if it is not possible to speak about standardized evaluation tests, it can only be said that some are more frequently used than others. The basic test is clearly to compare the synthetic word which was heard by the listeners participating to the test with the word actually synthesized to see if it is correct. However, many other more complex tests are available. Going back to the early works in this area, in Reference [72] the _Modified_; _RhymeTest_ (MRT) was proposed. MRT is based on the use of a rhyming words list that differ for the initial or final consonant; subjects must identify the right word within the set used as a stimulus and its output is the rate of success. MRT is a _closed_ test which allows to eliminate uncontrolled variations and it is very useful to test the intelligibility of particular aspect of the synthetic speech, as the initial or final CV transitions. The Diagnostic Rhyme Test (DRT), described originally in Reference [73] is a test for the evaluation of intelligibility based on the choice between two words whose initial consonants differ in a single distinctive trait, namely just a phoneme.