Evaluating the performance of speech technology systems

Present knowledge of spoken language (speech) combined with present status of technology already allows for the development of certain hardware and software systems that are capable to store and forward speech, to generate speech from text, to recognize the speaker who said something, and to recognize what has been said. Unfortunately, none of these speech technology systems is functioning perfectly well under all circumstances, so, scientists and system designers still have to work hard to improve the performance of their systems. Similarly, buyers of such products want Lo know what value they gel for their money. Both in the development phase and in the product evaluation phase, good methods are required to measure the performance in a, preferably standardized, diagnostic and comparative way. S ince the technology itself is rather new, also the methods to evaluate the performance of these systems are still under development, and constitute a research area in its own merits. We will report on the development of these evaluation methods and on the results achieved so far. 1 System Performance A speech technology system is generally developed in order to analyze, code, generate, or recognize speech. This is done with a certain goal in mind, such as the recognition of connected digits (e.g. product code) spoken over the telephone by many different speakers of English, or a spoken newspaper for visually handicapped people, or a low bit-rate speech coder for mobile radio use, or a speech interface in an office environment, or perhaps as a basic research tool to study how acoustic-phonetic knowledge can best be incorporated in a stochastically-based recognizer. In all these cases the optimal approach has to be chosen, and the system has to go through several steps for implementation, training, and testing. Each step requires informal evaluation by the designer before he can proceed to the next step. However, at certain points in the design phase, and most definitely at the completion of the system, a formal evaluation is required. This defines the performance in an unambiguous way. It is no longer sufficient then to say 'it sounds better than before', or 'the recognition score seems to be somewhat improved'. What is then required are well defined and generally agreed figures of merit. This may sound simple but it can become quite a problem. Take, for instance, a speech synthesizer or, more specifically, a system that pretends to generate intelligible and natural-sounding speech, from whatever text presented to it, in a specific language. For such a system there are many different aspects that should be considered, ranging from * Preliminary version of a paper presented for publication in The Encyclopedia of Language & Linguistics. IPA Proceedings 15, 1991 27 text interpretation and grapheme-to-phoneme conversion, via choice of speaking style, rhythm, voice characteristics, and intonation, to intelligibility at word and phoneme level. All these aspects jointly determine: the overall intelligibility and naturalness, as well as the adequacy for actual applications (announcements, database information, newspaper reading, spoken e-mail or telex messages), and the acceptability by the intended user (cooperative, experienced vs. untrained, naive). The situation is not always as complex as with the above example of a rule synthesizer. But even for a 'simple' word recognizer (specifically trained for one speaker, with only a small vocabulary of words, each word spoken in isolation), performance cannot unambiguously be defined. The percentage of correctly recognized words seems to be the most straightforward figure of merit. However, apart from the errors caused by substitutions (incorrectly recognized words), what to do then with insertions (word recognition without actual word input), deletions (no recognition at all, although a word was spoken), and rejections (not accepting a correctly spoken word)? How does the system react to non-vocabulary words or spurious sounds? Should one use live input only for testing, or would pre-recorded speech material be allowed? What are the environmental conditions (noise, competing speech, reverberation, telephone channel), how much training is allowed? Instead of counting words correct, one could argue that it is more relevant to specify how graceful the system recovers from an error, or what the actual throughput is in terms of time needed to perform a pre-defined task, for instance in comparison with another input modality, such as typing. One can imagine that moving from the relatively simple example of isolated word recognition to natural speech understanding in a dialog situation, further complicates a pe1formance definition. For instance, should one count words correct, sentences c01rect, or correct interpretation of meaning? As said before, apart from pe1formance of the final system, diagnostic assessment of a system under development is also of prime importance. Relevant aspects of the various subtasks then should be defined and their progress measured. This brings us to the scheme as presented in Table 1. Table 1. Aspects of performance and system development.