Reasoning About the User's Decoding of Presentations in an Intelligent Multimedia Presentation System

One of the goals in automated multimedia presentation planning is to tailor the presentations to the individual reader . Therefore, it can be useful for an intelligent multimedia presentation system to know whether the presentation techniques used will be understood. Reading multimedia presentations requires a decoding of the meaning conveyed through all relevant information carriers in presentations combin ing media such as text, drawings and animation. This decoding calls for inferences on the part of the reader. We discuss the kinds of inferences that are important in multimedia presen tations and the factors that influence their difficulty. A pro babilistic approach to the automatic assessment of the ease o f decoding a given presentation within the intelligent multi media presentation system W IP/PPPusing Bayesian networks is presented. An intelligent multimedia ( intellimedia) presentation system is an intelligent communication system that takes as input a collection of information to be conveyed and a communicative goal. From this input, it designs a presentation using various media such as text, graphics and animation, in such a way that the communicative goal is achieved. In the past decade, several approaches to the automated generation of multimedia presentations have been proposed. The clarity of the presentations has generally been taken into account in that the media allocation relies on rules or constraints based on common sense or psychological research findings—see, e.g., B OZ (Casner, 1991), C OMET (Feiner & McKeown, 1990), APT (Mackinlay, 1986), SAGE (Roth, Mattis, & Mesnard, 1991), WIP/PPP(André, 1995; Wahlster et al., 1993) and the approach presented by Arens, Hovy, & van Mulken (1993). However, that aspect of multimedia presentation understanding that seems to be one of the central criteria for the ease of decoding, namely the inferences that a presentation requires, has so far not received much attention. As with purely textual presentations (see, e.g., Graesser & Bower, 1990), in multimedia presentations not all information conveyed is presented explicitly: Often relations This research was supported by a graduate stipend from the German Science Foundation (DFG). between parts of a presentation stay implicit. It is exactly here that inferences are required on the part of the reader. Besides for reasons of brevity (e.g., Horacek, 1991) and being able to block false inferences (e.g., Marks & Reiter, 1990; Zukerman & McConachy, 1993), it is important for a presentation system to know what inferences a reader is required to make, because if it wants to know whether it will achieve its goal, it must consider what a reader must and can infer. This last case is the focus of the work described here. Its goal is to extend the intellimedia system P PP1 (Andréet al., 1993b; Andŕ e, 1995) with a component that can provide the system with knowledge so that it is able to decide whether a particular part of the presentation being generated will be decodable for a reader and further, how easily s/he will be able to decode it; and interpret evidence received from the reader through his/her feedback. PPP takes as input a communicative goal (e.g., the user should know how a certain switch is to be turned) and a series of generation parameters (e.g., space restrictions and media preferences). The output of the system is a multimedia presentation. P PPis an interactive system, that is, the user may direct the system as to how to continue the presentation (for instance, with less detail or with more elaboration on a particular part). Furthermore, s/he can ask questions about the object described in the presentation (e.g., how to perform particular requested actions, what functiona certain part of the depicted object has, and where particular parts arelocated). In the following sections, we discuss what decoding a presentation exactly involves, how inferences can be represented, and how the ease of decoding can be automatically assessed with the help of Bayesian networks. PPP is a follow-up project of WIP (Wahlsteret al., 1993). Unlike WIP, PPP is an interactive system. In the rest of the paper we will refer to PPPonly. Figure 1: Instructions for preparing a modem as output by P PP(with the parameter media preferenceset tographics). Decoding Multimedia Presentations Reading a presentation requires the reader to decode it so that s/he can construct a representation of its semantic and pragmatic meaning. As a working domain, we use instruction manuals for technical devices, such as those generated by P PP. The manuals consist of text and figures. When reading such a presentation, the reader has to deal with: textual information (titles, annotations, and main or accompanying text); graphical descriptions that depict real world objects; elements of the graphical language of instruction manuals, i.e., conventions such as explosion and cut-away views, and arrows indicating trajectories of movement; and layout, for instance, configurations of frames carrying meaning, i.e., relations between description parts. To extract information from all of these information carriers, processes of association and inference in the reader must take place. The reader must associate words and depictions with corresponding mental representations and link conventions of the graphical language with their meanings. Moreover, the reader must make inferences to relate the parts that a presentation consists of. In this work, an inference refers to a reasoning step that either dynamically or associatively 2 generates information on the basis of given information, so that a link can be made between mental representations, between mental representations and descriptions, or between parts of descriptions (see also Zukerman & McConachy, 1993; but see van der Meer & Schmidt, 1992, for a different definition of inferences). In multimodal presentations we can distinguish three major sources of inferences: Referring expressions in multimodal presentations require the reader to draw links between anaphoric expressions This distinction is related to that between strategicandautomatic inferences in McKoon & Ratcliff (1992). and antecedents, links between mental representations and descriptions, and links between mental representations (Andŕe & Rist, 1994). Semantic relationships between document parts often remain implicit and thus have to be inferred by the reader. For example, in the case of two document frames depicting the same object in different states, the reader may have to infer whether a temporal or an action-result relationship is meant. Graphical conventions have to be linked with their meanings. For example, an arrow expressing the trajectory of a movement would require an association that links the arrow with a mental representation of the movement trajectory. Representation of Inferences To represent inferences, we use the predicates introduced by André & Rist (1994), who present a model of referring to world objects with text and pictures. To describe the links that the reader has to make in understanding a referring expression, Andŕ e & Rist introduce the following predicate to represent world objects: (Coref <rep1><rep2>), which is used to express the fact that two representations <rep1> and<rep2> are representations of the same world object. To represent descriptions they introduce the relation (Encodes <device> <information> <context-space >), which expresses the semantic relationship between a graphical or textual <device> and the<information> the device is to convey in a certain specific context ( <context-space >); and (EncodesSame <p1> <p2> <context-space >) to express a cohesion relationship between two presentation parts<p1> and<p2>, which is satisfied if and only if <p1> and<p2> encode the same object. In the following section, the use of these predicates for the description of inferences is illustrated with an example. 1. The switch in the inset of the first frame corresponds to the smaller one in the main frame; both refer to the same world object: (EncodesSame p inset switch 1 p mainframeswitch 1 frame1) (Coref r1 u r2 u) (Encodes pinset switch 1 r1 u frame1) (Encodes pmainframeswitch r2 u frame1) 2. The encoded relation between the switch in the inset and the modem in the main frame is a part-whole relation: (Encodes pinset switch 1-p modem1 relation1 frame1) (part-whole relation1) 3. The encoded relation between the inset and the main frame is an action-result relationship: (Encodes pinset-pmainframe relation2 frame1) (action-result relation2) 4. The encoded relation between the three frames is a temporal sequence relation between the actions depicted in the individual frames: (Encodes psequence-of-frames 1 relation3 picture1) (temporal-sequence relation3) 5. The zigzag line in the third frame indicates that the light is on: (Encodes zigzag-line1 (on (light)) frame3) Figure 2: Some of the necessary inferences required by the modem instruction of Figure 1. The prefix “p” denotes a picture element; r1u and r2u are mental representations constructed by the reader/user.