Multimodal Systems, Resources and Evaluation

This paper considers multimodal systems, resources, and evaluation. We first motivate the value of multimodal information access with a vision of multimodal question answering and an example of content based access to broadcast news video. We next describe intelligent multimodal interfaces, define terminology, and summarize a range of applications, required corpora, and associated media. We then introduce a jointly created roadmap for multimodality and show an example of an open source multimodal spoken dialogue toolkit. We next describe requirements for and an abstract architecture of multimodal systems. We conclude discussing multimodal collaboration, multimodal instrumentation, and multilevel evaluation. 1. Multimodal Question Answering A long range vision of ours is to create software that will support natural, multimodal information access. As implied by Figure 1, this suggests transforming the conventional information retrieval strategy of keywordbased document/web page retrieval into one in which multimodal questions spawn multimodal information discovery, multimodal extraction, and personalized multimodal presentation planning. In Figure 1 the user of the future is able to naturally employ a combination of spoken language, gesture, and perhaps even drawing or humming to articulate their information need which is satisfied using an appropriate coordinated integration of media and modalities, extracted from source media. Figure 1. Ask Multimodal Questions, Get Multimodal Answers The inadequacy of the current document retrieval strategy most closely associated with web search engines is underscored by Figure 2. Figure 2 illustrates that while (normalized) computing power doubles every 18 months and storage capacity doubles every 12 months, the fastest changing area of infrastructure is optical networking, where network speed is doubling every 8 months. Coupled with the rapid deployment of wireless devices and infrastructure, the ability to support mobile, multimodal access is becoming reality. Figure 2. Acceleration of Infrastructure Growth 2. Broadcast News Access As a step toward multimodal question answering, we have been exploring tools to help individuals access vast quantities of non-text multimedia (e.g., imagery, audio, video). Applications that promises on-demand access to multimedia information such as radio and broadcast news on a broad range of computing platforms (e.g. kiosk, mobile phone, PDA) offer new engineering challenges. Synergistic processing of speech, language and image/gesture promise both enhanced interaction at the interface and enhanced understanding of artifacts such as web, radio, and television sources (Maybury 2000). Coupled with user and discourse modeling, new services such as delivery of intelligent instruction and individually tailored personalcasts become possible. Figure 3 illustrates one such system, the Broadcast News Navigator (BNN) (Merlino et al. 1997). The web-based BNN gives the user the ability to browse, query (using free text or named entities), and view stories or their multimedia summaries. For example, 0 2 4 6 8 10 12 14 16 18 0 6 12 18 24 30 36 Optical Network Speed doubles every 8 months Storage capacity doubles every 12 months Computing power doubles every 18 months Typed Query: Where was Ebola last reported in animals and humans on the coast of Gabon? NOW Multimodal Query: Where was Ebola last reported near this coast (spoken with gesture to map)? FUTURE BBC News. Friday, 11 January, 2002, 16:37 GMT Ebola in Gabon A World Health Organisation official, Gregory Hartl, expressed concern about 200 people who had been in contact with Ebola victims near Mekambo, a jungle town about 750 kilometres (465 miles) north-east of the capital, Libreville. There have been 34 confirmed cases reported so far, including 25 deaths 18 in Gabon and seven in the Republic of Congo. Another 200 people are being closely monitored. Fused, Tailored Multimodal Answers Multimodal Answer: 25 people died of Ebola in Gabon and Congo as of January 11 near the location shown here in the map Text Documents, not Answers Figure 3 displays all stories about the Russian nuclear submarine disaster from multiple North American broadcasts from 14-18 August 2000. This format is called a Story Skim. For each story, the user can view story details, including a closed caption text transcription, extracted named entities (i.e., people, places, organizations, time, and money), a generated multimedia summary, or the full original video. Figure 3. Tailored Multimedia News In empirical studies, Merlino and Maybury (1999) demonstrated (see Figure 4) that users enhanced their retrieval performance (a weighted combination of precision and recall) when utilizing BNN’s Story Skim and Story Details presentations instead of mono-media presentations (e.g., text, key frames, video). In addition to performance enhancement, users reported increased satisfaction (8.2 on a scale of 1 (dislike) to 10 (like)) for mixed media display (e.g., story skim, story details). Figure 4. Relevancy Judgement Performance with Different Multimedia Displays Figure 5. Video Annotation As illustrated in Figure 5, during system development we utilized annotation tools to markup a corpus of video for features such as program start/stop as well as commercial and story segments. Using this gold standard, we can apply hidden Markov models to automatically learn a cross modal statistical model for video segmentation and transition detection. Learned models can then detect such video elements as the start of commercial or the transition from a desk anchor to a reporter in the field (Boykin and Merlino 2000). Rapid creation of this multimodal corpora is essential. 3. Multimodal Interfaces Another vision is of intelligent multimodal interfaces that support more sophisticated and natural input and output, enable users to perform complex tasks more quickly, with greater accuracy, and improve user satisfaction. Intelligent multimodal interfaces are becoming more important as users face increasing information overload, system complexity, and mobility as well as an increasing need for systems that are locally adaptive and tailorable to heterogeneous user populations. Intelligent multimodal interfaces are typically characterized by one or more of the following three functions (Maybury and Wahlster 1998, Maybury 1999): Multimodal input – they process potentially ambiguous, impartial, or imprecise combinations of mixed input such as written text, spoken language, gestures (e.g., mouse, pen, dataglove) and gaze. Multimodal output – they design coordinated presentations of, e.g., text, speech, graphics, and gestures, which may be presented via conventional displays or animated, life-like agents. Interaction management – they support mixed initiative interactions that are context-dependent based on system models of the discourse, user, task and media. 1 See www.mitre.org/resources/centers/it/maybury/iui99 for an on-line tutorial on intelligent interfaces. 0 10 20 25 Average Time (seconds per story) 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1 A ve ra ge P er fo rm an ce (F -s co re = (P +R )/2 3 Named Entities All Named Entities Skim and Details Key Frame Skim Story Details