Multimedia interaction for the new millennium.

Spoken language processing has created value in multiple application areas such as document transcription, data base entry, and command and control. Recently scientists have been focusing on a new class of application that promises on-demand access to multimedia information such as radio and broadcast news. In separate research, augmenting traditional graphical interfaces with additional modalities of interaction, such as spoken language, gesture, or eye tracking, promises to enhance human computer interaction. In this address I discuss the synergy of speech, language and image processing, introduce a new idea for corpus based multimedia interfaces, and identify some remaining challenging research areas. MULTIMEDIA INFORMATION ON DEMAND Information on demand, the ability to provide information tailored to specific user needs, promises new capabilities for research, education and training, and electronic commerce (e.g., on-line information access, question answering, and customer service). Whereas significant commercial activity has focused on providing access to documents, web pages, and structured data sources, less attention has been given to multimedia information on demand. To achieve effective multimedia information on demand, however, requires a confluence of capabilities from several fields including image, speech and language processing, information retrieval, information extraction, translation, summarization, and presentation design. Over the past several years scientists have been exploring a range of systems to provide tailored, contentbased access to multimedia including text, imagery, audio, and video [19]. For example, by synergistically combining techniques for processing speech, language, and imagery, MITRE developed a sophisticated news understanding system, the Broadcast News Navigator (BNN) [21]. The web-based BNN gives the user the ability to browse, query (using free text or named entities), and view stories or their multimedia summaries (Figure 1 displays all stories about Diana on CNN Prime News during August-September 1997). For each story, the user is given the ability to view its closed caption text, named entities (i.e., people, places, organizations, time, money), a generated multimedia summary, or view the full original video of a story. The user can also graph trends of named entities in the news for given sources and time periods. For example, Figure 2 graphs the onset and abatement of stories on Princess Diana and Mother Teresa, 8-15 September 1997. Figure 1. Tailored Multimedia News Analyzing audio, video, and text streams from digitized video, BNN segments stories, extracts named entities, summarizes stories, and designs presentations to provide the end user with content-based, personalized web access to the news [21]. For example, within the video stream color histograms are used to classify frames and detect scene changes. In the audio stream, algorithms detect silence, speaker changes, and transcribe the spoken language. Finally, the closed caption stream and/or speech transcription is processed to extract named entities. This fully automated broadcast news system stands in contrast to the current method of manual transcription and summarization of broadcast news (e.g., via closed captioning services) which is expensive, error prone, and can result in dissemination delays. BNN has been integrated into a larger system called GeoNODE [12], which correlates named entities across stories to create story clusters and then partitions these clusters from a constructed hypergraph to automatically identify topics. Figure 2. Temporal Visualization of Named Entities Our analyses show that in key tasks such as segmenting stories, audio and imagery processing can enhance algorithms which are based only on linguistic cues (e.g., explicitly stated anchor welcomes, anchor to reporter handoffs, story introductions). For example, silence detection, speaker change detection, and key frame detection (e.g., black frames, logos) can improve the performance of text-only story segmentors. By modeling story transitions using hidden Markov models and learning the most effective combination of cross media cues [4], successive versions of the system have incrementally increased performance. As illustrated in Figure 3, the system’s version 2.0 performance over a range of broadcast sources (e.g., CNN, MS-NBC, and ABC) averaging across all cues is 38% precision and 42% recall. In contrast, performance for the best combination of multimodal cues rises to 53% precision and 78% recall. When visual anchor booth recognition cues are specialized to a specific source (e.g., ITN broadcasts that have more regular visual story change indicators) the performance rises to 96% precision and recall. Figure 3. BNN Segmentation Performance by Version Given properly delineated story boundaries, BNN is able to summarize the story in a variety of fashions. This includes extracting [1] the most significant named entities, extracting visual elements (e.g., keyframes) and/or summarizing the story text and creating from these elements a multimedia summary. We have integrated Carnegie Mellon University’s SPHINX-II speech system into BNN and have begun experiments in extracting named entities from transcribed stories. For example, Figure 4 shows a manual transcript of a segment of a story with human markup of locations (bold italic) and organizations (bold underlined). In contrast, Figure 5 shows an automated transcription of a news segment followed by automated markup of locations (bold italic) and organizations (bold underlined). Notice the errors in the automated transcript of omission of punctuation and content (e.g., “A”, “AND”) and substitutions (e.g., “EVELYN” for “CONSIDERING”, “TRIAL” for “PANEL”). Also note errors in the named entity identification in Figure 5 (“BLACK” is identified as a location, “UNITED STATES” and “CONGRESSIONAL BLACK CAUCUS” are missed). In the DARPA HUB-4 information extraction evaluation, the version of the system described in [26] achieved 71-81% accuracy on broadcast news speech data with a wide range of word error rates (WERs) ranging from 13% to 28%. The model achieved 88% accuracy on reference transcriptions with WER 0% (like Figure 4). Figure 4. Manual Transcription & Extraction Figure 5. Automated Transcription & Extraction Even when dealing with closed-captioned text, we face a 10-15% word error rate because of errors introduced during manual transcription. The word error rates for the best automated speech transcription systems (depending upon processing speed) range widely from 13-28% on WHITE HOUSE OFFICIALS SAY THE PRESIDENT IS CONSIDERING TWO STEPS: A NEW PRESIDENTIAL PANEL MODELED ON THE 1968 KERNER COMMISSION, WHICH CONCLUDED THERE WERE TWO SOCIETIES IN THE UNITED STATES, ONE BLACK, ONE WHITE SEPARATE AND UNEQUAL. A WHITE HOUSE SPONSORED CONFERENCE THAT WOULD INCLUDE CIVIL RIGHTS ACTIVISTS, EXPERTS, POLITICAL LEADERS AND OTHERS THE NEW HEAD OF THE CONGRESSIONAL BLACK CAUCUS SAYS THE PRESIDENT SHOULD BE DOING ALL THIS AND WHITE HOUSE OFFICIALS SAY THE PRESIDENT IS EVELYN TWO STEPS WHAT NEW PRESIDENTIAL TRAIL OF ALL ON THE NINETEEN SIXTY EIGHT KERNER COMMISSION WHICH CONCLUDED THE OUR TWO SOCIETIES IN THE UNITED STATES ONE BLACK ONE WHITE SEPARATE AND UNEQUAL WHITE HOUSE SPONSORED CONFERENCE THAT WILL INCLUDE CIVIL RIGHTS ACTIVISTS EXPERTS POLITICAL LEADERS AND OTHERS AND THE NEW HEAD OF THE CONGRESSIONAL BLACK CAUCUS OF THE PRESIDENT TO BE DOING ALL THIS AND MORE. 0 20 40 60 80 100