Human and Automated Indoor Route Instruction Following

Human and Automated Indoor Route Instruction Following Matt MacMahon (adastra@mail.utexas.edu) Department of Electrical and Computer Engineering; The University of Texas at Austin Brian Stankiewicz (bstankie@psy.utexas.edu) Department of Psychology; Center for Perceptual Systems Consortium for Cognition and Computation The University of Texas at Austin 1 University Station A8000 Austin, Texas 78712 USA Abstract that can typically be followed is remarkable. The current paper investigates how people give route instructions about indoor environments they have learned through navigation. We present a computational model of route instruction following called Marco and investigate the following three questions: 1. How do quality and style vary in route instructions? 2. How well does Marco follow route instructions and how closely does the model’s behavior correlate with human behavior? 3. Can Marco differentiate good versus bad instruc- tions, by both human performance and ratings? Humans possess the remarkable ability to give and follow natural language route instructions through large- scale spaces. In this process, a director describes the actions and observations along the route, recalling the environment’s topology, metrical layout, and visual features. A follower interprets these descriptions, navigating by applying the instructions to the possibly unfamiliar environment. Furthermore, followers must account for mistakes, ambiguities, and omissions in the route description. To study how instructions are written and followed, we collected 756 free-form route instructions from six participants for 126 routes in three virtual environments. A second group of participants and a computational model (Marco) followed these instructions. Humans successfully reached the destina- tion on 68% of the instructions and Marco followed 61% of the instructions. Marco’s performance was a strong predictor of human performance and ratings of individual instructions. Why are some route instructions reliable? Keywords Artificial Intelligence; Spatial Cognition; Natural Language Understanding; Cognitive architec- tures; Human experimentation; Symbolic computational modeling Knowledge representation Introduction Imagine while walking across a campus, a stranger approaches you to ask how to get to another location. The destination is not within sight, so you cannot simply point at the goal. Instead, you must reference your memory of routes between you and the goal. Once a route is selected, you need to access your knowledge of specific landmarks and distances to provide references for the follower . You translate this knowledge into a verbal description of the route. Remarkably, a short verbal description is often sufficient to guide a follower through an unfamiliar large-scale space. Despite directors’ best efforts, not all instructions are perfectly clear and reliable for reaching the goal. Often instructions contain ambiguous information (e.g. which tree is “the oak tree”), qualitative mistakes within the instruction (e.g. “turn right” where no right turn is possible) or metrical mistakes (e.g. “go forward 3 blocks” when the distance is 4 blocks). Because of these failings, the follower must treat the instructions as guidance, not as strict commands. With all the potential for miscommunication arising from the complexity of giving and understanding route instructions, the human ability to provide instructions Though a large literature examines route instructions, there is no consensus about what differentiates good instructions from bad. Vanetti and Allen (1988) found spatial ability had a larger effect in the accuracy of sub- jects’ described routes than verbal ability. Daniel et al. (2003) found “good”, “poor”, and “skeletal” instructions were differentiated by whether the proper action was associated with the proper landmark. Allen (2000) suggests descriptives clauses and action delimiters should be inserted at choice points and near the destination, instead of en-route. Lovelace et al. (1999) found good route instructions mentioned many landmarks along the paths, off the route, and at the choice points. Some of these studies rate route instructions sub- jectively (Vanetti and Allen, 1988; Lovelace et al., 1999; Tversky and Lee, 1999), but do not test if navigation success is affected. Others have participants follow a small number of participant- and experimenter-written instructions (Daniel et al., 2003; Allen, 2000). Our work implements a study suggested by Lovelace et al. (1999), where participants follow route instructions from different routes and directors in virtual reality. Computational models of route instructions Software systems that analyze or follow route in- structions can be distinguished by how they represent space. Freundschuh and Egenhofer (1997) survey a variety of spatial representation models and define broad categories based on (1) if the objects in the space are manipulable, (2) if the space requires locomotion to experience, and (3) the size, or scale, of the space. This work focuses on non-manipulable, large- scale spaces that cannot be experienced from any one