HANDS-ON SPEECH TECHNOLOGY ON THE WEB

The speech group at KTH has developed a number of speech technology tools for use in education of undergraduate students or researchers in the speech field. Many of these tools have been limited to a certain computer environment and the need for teacher guidance. The past years we have started developing a toolkit for spoken language technology that can be used over the Internet. In this presentation we want to focus on the possibiliti es to increase awareness of speech technology through both demonstrations and interactive experimental education, available on Internet. Examples of some of these applications will be demonstrated at the workshop. USING THE INTERNET FOR SPEECH TECHNOLOGY DEMONSTRATIONS Over the past three years the web has come to play an increasingly important role in our applications and demonstrations. The web is an important medium for publishing, as an application platform and as a knowledge base. In this presentation we will describe and present a number of examples where we have used the web for interactive speech technology demonstrations. One of the first efforts at CTT to use this novel vehicle for co-operative research was the creation of a web interface to our multi -lingual speech synthesiser [1]. This has made it possible to easily demonstrate the technology using any web browser anywhere. As an example, in 1996 we set up a demonstration of MIT ́s dialogue system GALAXY with our web-based speech synthesis as an integrated module. Several other speech R&D sites have set up similar faciliti es. We have collected some examples of speech synthesis presented on the web [2]. Software for web based speech analysis has also been developed [3]. The processing is done using speech analysis tools embedded in web pages. Students can record their own speech directly from the web page, view spectrograms, make measurements etc. The expectation is that the student in an explorative and interactive process will obtain a deeper understanding of the speech signal and its variabilit y We have developed plug-ins for Netscape for speech recognition, where the plug-in at the client side handles the audio recording and feature extraction, while servers at CTT perform the actual recognition and optionally the natural language processing and information retrieval. The response from the server could be anything ranging from the recognised text string to information in different forms, for example web-pages, images or synthesised speech. A modular spoken dialogue system has been developed [4], with our speech technology servers available over the Internet. The dialog system has been used in laboratory assignments in several speech technology classes in different universities in Sweden. The aim of this work has been to give students hands-on experience via a fully functioning spoken dialogue system as a teaching aid. Students were given some initial guidance on how to modify and extend the system but most of their work was unsupervised. We did an experiment running eight dialog systems simultaneously by about 20 students. In the next sections we will describe the different applications in more detail THE TMH SPEECH TOOLKIT We have recently developed a toolkit to alleviate the arcane art of constructing spoken language systems. This toolkit is based on the software technology in our existing spoken dialogue system WAXHOLM [5]. We have extracted and redesigned different components such as the ones for speech recognition [6], speech synthesis [7], visual speech synthesis [8], and parsing [9] and created Tcl language [10] modules from these. This has enabled us to take advantage of the rapid prototyping and development framework, which this language fosters and to create a toolkit for spoken language technology in the spirit of the ones created at OGI (CSLUsh) [11] and MIT (Sapphire) [12]. Our toolkit has empowered us to create new applications quickly and easily based on its modules using Tcl as a glue language and also to use the accompanying Tkwidget set for graphical user interfaces. Tcl is a rather simplistic language, with many shortcomings, but we use it only for system integration and user interfaces. The modules themselves are exclusively written in C and Java. Module interfaces are string based, which makes coding, testing, and debugging simple, but these interfaces could be limiting for future applications. The Broker Architecture We have developed an architecture for communication between programs on different computers [13]. It consists of one central server, the Broker, which relays function calls, results and error conditions between clients and servers over the Internet. It is designed to be simple and robust. All communication within the broker system is in text form to ensure portabilit y and aid in debugging. There is a debugging tool to watch all these transactions. Binary data, such as speech, is sent over separate TCP connections directly from producer to consumer. This architecture lets us keep our speech technology servers in house where they can run on powerful machines, and be updated at any time. We can also collect speech data regardless of where the applications are used. The Broker is written entirely in Java, and can therefore run on any modern computer. There are Java classes and Tcl code to support the writing of client and server modules, but the protocol is defined for any program that can communicate using the TCP protocol. We have tools that can display all communication between modules to facilit ate debugging. The Broker has been successfully used with about 50 simultaneous connections, and also between an application in Washington D.C. and our servers in Stockholm. The full source code for the Broker system is available "as is" at http://www.speech. kth.se/proj/broker. SPEECH ANALYSIS ON THE WEB In our courses on speech technology we have an introductory section on basic phonetics and speech analysis. For this section we have developed a set of exercises in which students analyse their own speech in various ways. These exercises are accessed through web pages [14] in which simple speech analysis tools have been embedded as applets. In this way we have been able to supply these exercises to students both working in our lab, at Linköping University and from their home PC ́s. The analysis tools were created using Tcl/Tk and an extension module developed at our lab. This makes it possible to run these tools from Netscape on three different Unix platforms and from Netscape or Internet Explorer on MS Windows. The tools can also be executed outside and independently of a web browser, but in this case they first must be downloaded and installed. The big advantages of using a web browser as a platform is that all i nstallation issues are solved and instructions and other useful information can accompany the tools in an easily accessible way. A screen-shot of one of the exercises is shown in Figure 1. The exercises covered measurements of vowel formant frequencies, comparisons of speakers and speaking styles, Swedish word accent, and phonetic segmentation. Figure 1. A screen-shot of one of the speech analysis exercises running inside Netscape navigator. SPEECH SYNTHESIS LAB We have developed a graphical interface to our text-to speech system [7]. This system uses a synthesis server via the broker architecture. The synthesis server that runs on our HP-UX-system is for the moment set up with five of the ten languages available in our text-tospeech system. With this set-up our speech synthesis can be used by clients anywhere in the world. The Swedish version uses a number of lexicons, including a large name lexicon constructed as part of the Onomastica project [15]. The simplest graphical interface to the synthesis server only has a text entry, a transcription field and buttons to create and play synthesised speech. If the user wants to change the prosodic pattern in the sentences, they can switch to a larger text input window where the text can be tagged with different stress levels, indicated by different colours, see Figure 4 in the section about the dialog system. Texts that have been modified in this interface can be sent to the most advanced interface where the user can edit all parameters in the formant synthesiser, see Figure 2. Figure 2. Combining two modules (speech synthesis generation and spectrogram) in one system. In this interface it is possible to select which parameters to display an edit. The parameter trajectories are easily changed with the mouse and synthesis can be generated automatically whenever a control point is released. The system can also handle the parameters that are used for the multi -modal synthesis, simply by choosing Use face for Playback on the meny. A new window with the animated 3D-face will appear as well as new parameters that are used for the face animation. In this set-up the user can experiment with visual cues in the multi -modal speech synthesis, such as eyebrow raise at important words, or to indicate questions. It is also possible to combine the synthesiser module with speech analysis module described in the previous section, as is shown in Figure 2. The software is a new platform for speech synthesis experiments in education, and also in research. A SYSTEM FOR TEACHING SPOKEN DIALOGUE SYSTEMS TECHNOLOGY Traditionally, spoken dialogue systems have required high expertise to design and develop. The resulting applications have been large and complicated and not always easy to modify and extend or even comprehend. Thus, teaching in the subject of spoken dialogue systems and related technologies has mostly been done in lecture format with video taped demonstrations of actual systems. Live demonstrations are typically conducted by a well -behaved PhD student, who knows which questions to ask. Students have mostly been kept at a safe distance. The aim of this work has been to put a fully functioning spoken dialogue system into the hands of the st