Questions, Pictures, Answers: Introducing Pictures in Question-answering Systems

We present the Dutch IMIX research programme on multimodal interaction, speech and language technology. We discuss our contributions to this programme in the form of two research projects, IMOGEN and VIDIAM, and the technical integration of the various modules developed by IMIX subprojects to build a demonstrator that shows the project’s aims and achievements. In this paper we pay special attention to the role that images could play in iterative question answering. 1 IMIX: multimodal interactive question answering This paper describes research performed at the Human Media Interaction group of the University of Twente, in cooperation with the Communication and Cognition group at Tilburg University, in the framework of the IMIX programme. IMIX is a research programme in the field of Dutch speech and language technology funded by the Netherlands Organisation for Scientific Research (NWO). It started in 2004 and brings together research on question answering (QA), dialogue management, speech recognition and generation, and information presentation. While early research in the field of QA concentrated on answering factoid questions 1 one by one in an isolated manner, recent research extends QA in several new directions. One of these directions is towards developing iterative QA or QA dialogue systems that allow users to ask follow-up questions and that can help users to clarify questions (Kato et al., 2006). Several projects that are part of the ARDA AQUAINT program concern scenario-based QA, the aim of which is to build systems that can handle non-factoid, explanatory, analytical questions posed by users with extensive background knowledge (Williams et al., 2004). Most of these systems interact with the user by means of text only. LIMSI’s Ritel system is a spoken dialogue system for open domain QA, but it only deals with factoid questions (Galibert et al., 2005). The IMIX programme further extends iterative QA in the direction of multimodal QA. The system is primarily text-based, but it allows users to use speech input as well as text input, and the answer is presented in the form of text, speech and pictures. The next step is to allow users to refer to the pictures in the answer presentation when asking follow up questions. How can QA, dialogue, and multimodality be combined? How can such a multimodal iterative QA system help users find the information they need? A number of Dutch research groups participate in the IMIX programme, collaborating to build a common demonstrator. This demonstrator is meant as a vehicle to prove that the research carried out in the individual projects can be applied and integrated in the context of a QA system. The IMIX demonstrator is an iterative multimodal QA system aimed at providing general medical information. The system will be able to deal with more complex questions than simple factual ones, and it will be able to engage in a simple dialogue with the user aiming at obtaining a better understanding of the question or at a more equal level of communication between the user and the system. Our Human Media Interaction group participates in two of the IMIX research projects: IMOGEN (Interactive Multimodal Output Generation), together with Tilburg University, and VIDIAM (Dialogue 1 Factoid questions are those for which the answer is a single fact. For example “When was Mozart born? ” “How tall is the Eiffel Tower? ” Management and the Visual Channel). A common interest of these two projects is the use of images in QA dialogues. Multimodal answer presentation, combining text, speech and pictures is the focus of IMOGEN, while VIDIAM researches the ways that users could interact with the system by pointing at pictures in their follow-up utterances, and how the dialogue manager could meaningfully react on these multimodal utterances. This paper is structured as follows. First, we provide a general overview of the different projects within the IMIX programme by discussing the functionality and the architecture of the IMIX demonstrator in section 2. Then, in sections 3 and 4 we zoom in on the respective contributions of the IMOGEN and VIDIAM projects. In section 5 we discuss the role that images could play in iterative question answering, and the research that has been done so far on this topic within IMOGEN and VIDIAM. We outline our future work in this area in section 6, and end with some conclusions in section 7. 2 The IMIX demonstrator: functionality and architecture The IMIX demonstrator is a system that helps users finding information they need in the medical domain. The system is presented as an Information Search Assistant, i.e., more like a librarian who helps users to find the medical information they need than a medical expert (see op den Akker et al., 2005). This Information Assistant is personified in the form of a Dutch-speaking version of the Rutgers University Talking Head (RUTH). 2 Regarding the nature and difficulty level of the questions it is supposed to handle, particular choices have been made. We try to provide a coverage that is sufficient to answer the information needs of real, non-expert users. We call the type of questions that we cover “encyclopaedic questions”. These are general medical questions that do not require expert medical knowledge (this excludes diagnostic questions). Nevertheless, correctness of answers is less well-defined and harder to measure than with factoid questions. Our system typically answers with a piece of text about 1-3 sentences long. The length and level of detail of a correct answer depends on the information need of the user, which cannot be precisely known beforehand. Enabling the user to give feedback on her information need is one of the purposes of our dialogue system. Although IMIX focuses on a closed medical domain, many of the techniques used are also applicable to open domain QA. This is especially true of the dialogue management module. It uses only little domain knowledge, which makes it easy to generalise to other domains. Figure 1: Global architecture of the IMIX demonstrator. A global architecture of the IMIX demonstrator is shown in Figure 1. The user can enter a question to the system in Dutch, either by typing or speaking. Both input modes can be combined with pen input (pointing or drawing). The speech recognition module of the demonstrator is provided by the NORISC project (University of Nijmegen), which focuses on advanced methods for Dutch speech recognition (Hämäläinen et al., 2005; Han et al., 2005). Pen input is dealt with in the VIDIAM project (this paper and van Schooten, these proceedings). The dialogue manager analyses the user’s input and determines how to deal with it. If the user has posed a domain question, this is sent to QA for answering. As the dialogue proceeds the user can also react with other utterances than questions, such as an acknowledgment of an answer presented by the system, or an indication that she doesn’t understand the answer. This implies that not all follow-up utterances are domain questions that can be sent to QA. In addition, follow-up questions may contain anaphoric references that need to be rewritten to form a self-contained question that can be handled by QA. Within IMIX, two projects deal with dialogue management. PARADIME (Tilburg University) focuses 2 http://www.cs. rutgers.edu/~village/ruth/ Audiovisual Speech Synthesis