An Integrated Evaluation of Perception, Interpretation, and Narration

In this paper, we describe our efforts to create an evaluation tool to aid in the development of artificial intelligence systems that integrate perception, reasoning, and language abilities. Based on an early and influential study by social psychologists Fritz Heider and Marianne Simmel, we created 100 short movies depicting the motions of two triangles and a circle around a box with a hinged opening. For each movie, we provide quantitative information about each object's trajectory, a formal description of the actions that can be perceived in each object's behavior, a formal interpretation of the social situation that is depicted, and a short English narration of the interpreted events. Perception, Interpretation, and Narration In an early and influential study, social psychologists Fritz Heider and Marianne Simmel (1944) presented subjects with a short animated film depicting the motions of two triangles and a circle around a box with a hinged opening. Asked to describe what they saw, subjects responded with coherent narratives of the social interactions among three characters, with interpretations of the motivations, intentions, and emotions of the anthropomorphized shapes. From an artificial intelligence perspective, these subjects engaged in a particularly interesting suite of cognitive tasks, effortless for people but challenging for today's technologies. What would it take to build an artificial intelligence system that could watch a movie in the style of Heider and Simmel, and generate a textual narrative indistinguishable from one written by a human? We imagine that successfully building such a system would require the integration of three emerging technologies from three different areas of artificial intelligence research. First is the problem of perception. The system must recognize the sequence of actions intended by the animator in the trajectories of abstract shapes. In the original HeiderSimmel film, the larger of the two triangles jabs at the Copyright © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. smaller one, pushing it up against the side of the box. The circle and the smaller triangle kiss and dance after being separated for some time. Procedurally, the task is one of segmentation and labeling of time-series data, and could be approached in much the same manner as seen in pen-based handwriting recognition systems and video-based gesture recognition systems. Second is the problem of interpretation. The system must explain the observed actions in anthropomorphic terms, ascribing motivations, intentions, and emotions to each of the characters. In the original Heider-Simmel film, the circle cowers in fear as the larger triangle corners it inside the box. The larger triangle becomes furious when the two other character escape, and destroys the box in a fit of rage. Procedurally, we see this interpretation task as one of logical abduction, where a knowledgebase of human sociology and psychology is used to find a parsimonious explanation of the observed actions. Third is the problem of narration. The system must generate a coherent discourse to communicate its interpretation of the perceived actions in a natural, humanlike style. By Heider and Simmel's report (1944), subjects authored narratives rich with mentalistic phrases: the girl hesitates, she doesn't want to be with the first man, the girl gets worried, is still weak from his efforts to open the door, they finally elude him and get away, he is blinded by rage and frustration. Procedurally, the task is that of discourse generation, where the relational structure of the interpretation informs the discourse structure, and the semantics of the interpretation inform the lexical choices. An Integrated Evaluation Individually, each of the tasks of perception, interpretation, and narration pose difficult artificial intelligence challenges, although none seem insurmountable given recent progress in their respective research areas. However, the integrated task presents new challenges and new questions. Recent work in segmentation and labeling of time-series data has focused on supervised approaches that output probability distributions. How are these distributions to be successfully integrated with the formalisms used in logical abduction? Moreover, how can the results of the interpretation process be used to adjust these probability distributions, such that the interpretation influences what is perceived? Recent work on discourse generation in computational linguistics has pursued a strongly statistical approach, realizing the most probable surface text given the underlying representation. How is this process changed when the underlying representations are structural interpretations, containing both informative and commonsense assertions? To tackle both the individual and integrated research challenges of perception, interpretation, and narration, we developed a new evaluation tool based on the original Heider-Simmel film. This evaluation tool consists of 100 short movies in the style of Heider and Simmel. To support the development of an integrated processing pipeline, this collection includes gold-standard annotations for each of the intermediate processes needed to translate these movies into coherent textual narratives, described below.