Conversational interaction for semantic access to multimedia information

Effective interaction is essential for effective information retrieval. The entire process of retrieval interaction should be organized in a way that allows the user to concentrate on her primary goals, i.e., clarifying vague information needs, finding appropriate search strategies, and assessing the relevance of retrieved information. Accordingly, an intelligent retrieval system must be capable of interpreting and negotiating ambiguous information needs interactively with the user. The MIRACLE system introduced in this chapter offers semantic access to a large, experimental multimedia information base. Both the retrieval engine and the dialogue manager employ abductive reasoning to generate plausible interpretations of ambiguous user requests/ queries and to create coherent interaction options based on a comprehensive dynamic dialogue model. As opposed to most state-of-the-art retrieval systems, MIRACLE supports the user actively on both the pragmatic and the cognitive level. 1 Why Conversational Interaction? Current multimedia information systems employ a variety of advanced data representation, indexing, and retrieval methods, but mostly apply less elaborate – often simplistic – models of human-computer interaction. Usually, contentbased retrieval methods are employed to induce similarity relationships among the available information objects, which can then be inspected by means of browsing operators. However, as the information systems and the human-computer interactions grow increasingly complex, the entire process of information search, retrieval, and relevance assessment needs to be actively supported by easy-to-use multimodal user interfaces. Many users of information retrieval (IR) systems have initially only vague information needs and often encounter problems formulating them precisely. 2 Conversational Interaction for Semantic Access to Multimedia Information Vague and ambiguous information needs are typical of multimedia retrieval, but occur in text retrieval as well. However, whereas it seems to be acceptable that users discard non-relevant textual items as long as highly relevant documents are shown at the top of the ranking lists, the relevance assessment required in multimedia retrieval imposes an even higher cognitive load on the user. Some data types, e.g., audio and video sequences, may be accessible using standard retrieval methods (based on their speech components, cf. Jones et al., this volume), but they still differ from textual documents, because they cannot easily be skimmed like texts or pictures. In addition, the tacit assumption underlying most text retrieval systems – that relevance can be reduced to topical similarity, which, in turn, is based on the (multiple) occurrences of index terms – has not yet found a convincing counterpart for non-textual media. Instead, there is growing evidence that retrieved data items (including texts) are assessed according to multi-dimensional relevance scales which express the specific needs of users in their current situation. Hence, we claim that clarification of information needs through interaction with the system is as important a task as the employment of appropriate indexing and retrieval mechanisms. As this type of interaction includes metacommunication about dialogue goals and strategies, it can be modeled as conversational interaction between user and IR system. From a user-centered point of view, intelligent multimedia retrieval systems should be capable of supporting the user in the following tasks: • Queries must be disambiguated to assign appropriate retrieval methods; • different interpretations of the query must be managed, and the relevant information objects must be determined for each of them; • the information objects must be presented in such a way that the user can relate them to her query and to each other; • unexpected problem situations and dialogue strategies should be negotiated and clarified interactively with the user. The first two issues can be tackled using semantic representations of information objects, linking them to the objects by content-based retrieval operators, and combining them with techniques for query disambiguation (e.g., abduction). Whereas the third issue requires powerful presentation and user guidance facilities, the fourth addresses the pragmatic aspects of information-seeking dialogue. In this chapter we will discuss our approach to combining these levels by intertwining multimedia IR and dialogue planning. Stein, Gulla, Müller, and Thiel 3 The next section gives a brief account of related research in various fields that are relevant for our work. In Section 3 we introduce the system architecture of MIRACLE, which integrates a logic-based retrieval engine and dialogue manager as its major components. As both employ abductive reasoning, we discuss the use of this inference technique from the retrieval point of view in Section 4, and from the dialogue planning perspective in Section 5, analyzing one typical example taken from MIRACLE. A brief summary and outline of future work are given in the conclusions of the chapter.