Meeting Recorder

Despite recent advances in automatic speech recognition (ASR) technology, successful ASR applications tend to be limited to telephony, command-and-control functions, head-worn microphones, small vocabularies, and/or simple grammars. Recording and processing spontaneous, human-to-human conversations in natural settings is a far more challenging task. Face-to-face meetings are not only challenging because of acoustic uncertainty, but they also contain very rich content. In contrast to typical ASR corpora, meetings contain a vast amount of overlapping speech, interrupts, false starts, laughter, and other interesting linguistic, acoustic, and prosodic features. To address some of these issues, we have begun research on the recording and processing of meetings. In this paper, we will describe the Meeting Recorder project, including possible applications, corpus collection, and progress to date. 1 Why Record Meetings? 1.1 Challenging Speech Recognition Firstly, natural human-to-human meetings provide a new, very challenging domain for research into automatic speech recognition (ASR). Although it is similar in spirit to other conversational corpora (such as Switchboard [1]), meetings pose an even greater problem for ASR [2]. The acoustic environment in a real meeting tends to be quite variable. Background noise such as fans, door slams, and paper rustling can all contribute to the acoustic background. Reverberation and echo can also be a significant problem in real rooms with desktop microphones. Many sections of real meetings contain large amounts of overlap − people end each other’s sentences, interrupt, encourage (“uh huh”), laugh, and so on. Not only do ASR systems perform very poorly on overlapped speech (the “cocktail party” problem), but overlap also makes traditional segmentation into speaker-disjoint regions somewhat problematic. For more details on segmentation, see section 4.1, below. As part of the corpus collection process at ICSI (discussed in more detail in section 4, below), we record real meetings that take place regardless of the Meeting Recorder project. As such, they appear to be quite natural. They contain all the artifacts of spontaneous speech seen in other corpora, including false starts, corrections, filled pauses, and disfluencies. Furthermore, since most of the recordings are of regular weekly meetings, the participants often know each other quite well. In many cases, this leads to highly informal conversational styles, with many interrupts, asides, back channel communications, jokes, etc. As compared to prepared speech or read speech, this style of speech is quite difficult for traditional ASR systems to recognize. Because of all these difficulties, we expect very poor initial results for meetings. As such, the corpus provides an excellent arena for research into robust ASR. See section 5 for some very preliminary results. 1.1.1 Digits In order to provide a simpler task, and to isolate acoustic issues from other effects, we also record digit strings as part of the corpus. At the start or end of each meeting, the participants are given a sheet of paper with about 20 lines of digits. Each line contains from one to ten digits spelled out in English (so that “zero” and “oh” can be distinguished). They are asked to read them in order, with a short pause between lines. The digits strings themselves are taken directly from an existing digit corpus called Aurora [3], although the recording environment in Aurora is quite dissimilar. The digits task is much easier than the general task for a variety of reasons. There is no overlap, since each participant reads the digits separately. Because it is read speech, there are fewer disfluencies and pauses. Finally, the number of words in the vocabulary is merely 11, rather than the 65000 that are typical of a dictation task. 1.2 Meetings are Interesting Meetings are also interesting in their own right. In addition to the acoustic characteristics described above, there is also interest in extracting prosodic information, perhaps to detect salient parts of the meeting. Sorting by importance may aid information retrieval. It would also be extremely useful to detect specific parts of a meeting such as summary information, progress reports, action items, and so on. Finally, social structures in meetings may be derivable from detailed information present in the corpora who speaks the most? Who interrupts whom? When does laughter occur? 2 SpeechCorder A primary goal of Meeting Recorder is to provide a system that can search the audio record of a meeting using spoken or written queries, either in real-time during a meeting or offline. Combined with a note taking application, such a system could replace written notes and tape recordings with a much more useful and interactive tool. A related project underway at ICSI, called SpeechCorder, aims to provide such a device on a portable computer. In this section, we will discuss only the information retrieval application itself. Details on the hardware and software of SpeechCorder can be found in [4]. 2.1 Why Not Written Notes? • Unreadable Handwriting: Often, written notes are illegible. This becomes even more of a problem as time passes. Notes that were comprehensible when written become indecipherable after a few weeks. • Detracts From Listening: It can be very difficult to take notes and pay close attention to the meeting at the same time. Many people find that they can take notes or they can listen closely, but not both. • Incomplete Record: It is almost impossible to record a meeting in its entirety. Frequently, important points and data are missing from the written record. • Hard to Search: Written notes provide no support for searching and indexing. Finding particular information is very difficult. 2.2 Why Not A Tape Recorder? • Hard to Annotate: With written notes, adding arbitrary annotations (underlining, circling, diagrams, doodles) is trivial. It is much more difficult to provide synchronized annotations with a tape recorder. • Playback Can Be Disruptive: One cannot play back a tape recording during a meeting without disturbing the other participants. Also, one cannot play back and record simultaneously. • Too Much Data: Recording an entire meeting via tape produces a very large amount of extraneous data. • Hard to Search: As with written notes, a tape recording of a meeting cannot easily be indexed and searched. 2.3 Hypothetical Meeting Recording Example Figure 1 shows a hypothetical Meeting Recorder screen shot. The image demonstrates some of the potential userinterface ideas for correction and annotation of the transcript. As participants of the meeting speak, a possibly inaccurate transcript appears in real-time. Speaker change boundaries are detected and shown (in this case, just by numbering). Since the speech recognizer often can produce a measure of how confident it is with its guess, the user-interface could highlight potentially poor transcriptions. When the recognizer makes an error, the user can quickly correct it by clicking on the incorrect word. The recognizer generates a popup box of alternatives from which the user can select the correct word. Also shown is a text annotation entered by the user. 1 The non-lexical parts speech such as pitch, speaking rate, intonation, etc. Figure 1 – Hypothetical SpeechCorder Screen 3 SpeechCorder Requirements For SpeechCorder to be useful, it must meet a number of challenging requirements. It must: 1. Record meetings in natural settings. 2. Support multiple speakers. 3. Work stand-alone. 4. Support indexing and searching. 3.1 Record Meetings in Natural Settings One of the primary design goals of SpeechCorder is to record real meetings in real settings. We would also like to include the ability to record impromptu meetings. The speech recognition must therefore work in uncontrolled acoustic environments, as described above. The vocabulary must be large enough to cover the domain of the meeting, and it must be robust to foreign accents. 3.2 Support Multiple Speakers Obviously, meetings have multiple participants. The speech recognition system should not only transcribe the speech of different people, perhaps talking at once, but it also should note when the speaker changes. It would also be convenient if it could record not only the speaker change, but also the identity of the speaker. This would allow searching by speaker as well as by the audio contents of the record. Speech recognition systems can improve their accuracy by adapting to a particular user. Allowing the system to store and exchange user profiles among meeting participants could dramatically improve the quality of the transcript. 3.3 Work Stand-alone If we want to support impromptu meetings in uninstrumented environments, it is necessary for SpeechCorder to be portable. Although it is certainly possible to use a hand-held computer as a terminal using a wireless network, we feel that a self-contained solution is better in the long run. The terminal/main-frame model has more components to fail the terminal, the network, the wireless link, the mainframe, and the infrastructure. There is also the question privacy, which is much easier to address with a self-contained unit. The PC revolution has also shown us the utility of personal compute power. 3.4 Support Indexing and Searching As described in section 2 above, we plan to provide for searching of the record both by text content and by speaker. Since the actual audio will be stored, the user can play back the portion of the meeting that matches his criteria. We will support both textual and spoken queries. Note that, since the audio record is stored, the transcript need not be perfect. It need only be good enough so that queries match where users expect them to match. For referring to the actual content, the user can play back the audio. In addition, the speech recognizer can output not just the most likely transcript, but also a list of the top f