A collaborative, non-invasive hybrid semantic localization and mapping system (HySeLAM)

Mobile robotics which presupposes interaction with humans requires new paradigms of the concept of intelligence. The robot’s ability to understand and perform tasks such as: “Robot, go to the office and bring me the pen that is on the table”, is a requirement. Most approaches to the localization and mapping layer are not in tune with the higher layers to solve this kind of task. The robot’s ability to reason about what it senses and knows, as well as to acquire and share knowledge when interacting with a human voice, are capabilities required for the success of this cooperation. Unlike traditional SLAM (Simultaneous Localization and Mapping) methods, which only interpret sensor information at the geometric level, these capabilities require an environment map representation similar to that of humans. This work aims to give an answer to the question How can a SLAM approach be extended in order to include the human into the mapping process? and for that it explores and investigates a hybrid approach (metric and semantic) for the localization and mapping layer, in order to reach a human-like environment map representation (knowledge). This will allow for the development of a new strategy that will enable the robot to: complete and build this knowledge through human interaction; associate sets of characteristics of observations to words, which are, in turn, associated to objects and places; and reason at a semantic level about this knowledge and its own observations. In this thesis, a novel semantic framework extension for SLAM is proposed in order to extend the geometric mapping into topological and semantic mapping. The name of this framework is HySeLAM, which stands for Hybrid Semantic Localization and MApping. This framework formalizes two new layers (topological and semantic) connected to the occupancy gridmap in order to describe places and objects and the relation between them. The semantic mapping process happens when the knowledge acquired from the human description is merged into these topological and semantic maps. In order to translate the occupancy grid-map into the augmented topological map, the formalization of an approach to discretize a grid-map into a topological map was required. This approach was implemented with the name Gr2To, which stands for gridmap to topological map tool. This tool abstracts the gridmap with delimited places and the connectivity between places. The obtained topological map from Gr2To is not complete and was not validated by a human. To solve this an approach was formalized to merge a description given by a human into the internal augmented topological map. This approach was implemented with the name TopoMerg, which stands for topological merging tool. A semantic localization procedure was also proposed, which locates the robot in a space defined by human words instead of a geometric space, as in a conventional SLAM. This approach was implemented with the name SeloViS. Using only a visual signature, which is obtained from a new global image descriptor, this procedure is highly accurate in recognizing a given place (more than 80% in the testing data). With this procedure, the HySeLAM topological layer is able to recognize a place at a glance as humans, to detect problems in the conventional SLAM and to help