Context-aware Spoken Language Understanding for Human Robot Interaction
Authors: Andrea Vanzo, Danilo Croce, Roberto Basili, Daniele Nardi (Sapienza Università di Roma, Università di Roma Tor Vergata, 2016)
Topic: Chatbots and Dialogue Systems

Abstract
Robots operate in specific environments and the correct interpretation of linguistic interactions depends on physical, cognitive and language-dependent aspects triggered by the environment. In this work, we present LU4R - adaptive spoken Language Understanding 4 Robots, a Spoken Language Understanding chain for the semantic interpretation of robotic commands, that is sensitive to the operational environment. The system has been designed according to a Client/Server architecture in order to be easily integrated with the vast plethora of robotic platforms.

Introduction
End-to-end communication in natural language between humans and robots is challenging for the different cognitive abilities involved during the interaction. As an example, for a robot to react to a command like ``take the book on the table'', a number of implicit assumptions should be met. First, at least two entities, book and a table, must exist in the environment and the speaker must be aware of such entities. Accordingly, the robot must have access to an inner representation of the objects, e.g., an explicit map of the environment. Second, mappings from lexical references to real world entities must be developed or made available. In this respect, the Grounding process AUTHOR links symbols (e.g., words) to the corresponding perceptual information. Hence, robot interactions need to be grounded, as meaning depends on the state of the physical world and the interpretation crucially interacts with perception, as pointed out by psycho-linguistic theories AUTHOR. To this end, the integration of perceptual information derived from the robot's sensors with an ontologically motivated description of the world has been adopted as an augmented representation of the environment, in the so-called semantic maps AUTHOR. In this maps, the existence of real world objects can be associated to lexical information, in the form of entity names given by a knowledge engineer or spoken by a user for a pointed object, as in Human-Augmented Mapping AUTHOR. While SLU for Interactive Robotics have been mostly carried out over the only evidences specific to the linguistic level (see, for example, AUTHOR), we argue that such process should be context-aware, in the sense that both the user and the robot access and make references to a shared environment. For example, in the above command, ``taking'' is the intended action whenever a book is actually on the table, so that ``the book on the table'' refers to a single argument. On the contrary, the command may refer to a ``bringing'' action, when no book is on the table and the book and on the table correspond to different semantic roles. In this paper, we present LU4R an adaptive spoken language understanding chain for the automatic interpretation of robotic spoken commands that is coherent with the above assumptions. The resulting chain is based on the approach proposed in AUTHOR that allows to produce interpretations that are consistent with (i) the world (with all the entities composing it), (ii) the Robotic Platform (with all its inner representations and capabilities), and (iii) the linguistic information derived from the user's utterance. LU4R is fully implemented in Java and is released according to a Client/Server architecture, in order to decouple the chain from the specific robotic platform that will use it. It receives as input one or more transcriptions of a spoken command and produces one or more linguistic predicates reflecting the actions intended by the user. Predicates, as well as their arguments, are consistent with a linguistically-motivated representation and coherent with the environment perceived by the robot. The rest of the paper is structured as follows. Section~sec:arch provides an architectural description of the entire system, as well as an overall introduction about its integration with a generic robot. In Section sec:exp we demonstrate the applicability of the chain in the interpretation of commands in English and Italian.

Conclusion
In this paper, we presented LU4R, an adaptive SLU processing chain focused on the interpretation of commands in the Mobile Service Robotics domain. The proposed solution relies on Frame Semantics and Distributional Lexical Semantics to support example-driven machine learning algorithms that map individual sentence transcriptions to meaningful commands. Statistical learning, i.e. \svmhmm, is applied by transforming the interpretation process into a cascade of sentence annotation tasks. The use of Frame semantics enables the reuse of large repositories of examples (i.e. AUTHOR), supporting the recognition of up to more than 1,000 different semantic frames, currently defined in the FrameNet database. The robustness of the sentence interpretation, as measured in this paper, is rather good, where language variability is tackled by relying on distributional lexical models that generalize semantics for large vocabularies, well beyond the training set dictionaries. Moreover, even though LU4R is completely decoupled from the Robotic Platform, the final interpretation is made dependent on the robot's environment, by designing perceptual knowledge through feature modeling. Perceptual knowledge is derived from the robot's semantic map and translated into feature values. The corresponding space allow to synthesize information about existence and position of named entities, useful to disambiguate predicates and role assignment. The results gathered during repeated empirical investigation campaigns confirm the effectiveness of the proposed tool and, in particular, the benefits provided by the injection of perceptual Knowledge into the understanding (i.e. labeling) process. LU4R is thus an effective example of grounded language learning framework, whereas predicates and semantic roles are acquired through an integration between linguistic (e.g. lexical and grammatical) properties and perceptual information (e.g. distances between entities in a map): this makes LU4R an interesting topic for future research, such as the extension of command interpretation process to complex interaction patterns, such as in interactive question answering or dialogue.