Query: deep learning based multimodal perception in autonomous robots

title: Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
abstract: Much of the literature on robotic perception focuses on the visual modality. Vision provides a global observation of a scene, making it broadly useful. However, in the domain of robotic manipulation, vision alone can sometimes prove inadequate: in the presence of occlusions or poor lighting, visual object identification might be difficult. The sense of touch can provide robots with an alternative mechanism for recognizing objects. In this paper, we study the problem of touch based instance recognition. We propose a novel framing of the problem as multi modal recognition: the goal of our system is to recognize, given a visual and tactile observation, whether or not these observations correspond to the same object. To our knowledge, our work is the first to address this type of multi modal instance recognition problem on such a large scale with our analysis spanning 98 different objects. We employ a robot equipped with two GelSight touch sensors, one on each finger, and a self supervised, autonomous data collection procedure to collect a dataset of tactile observations and images. Our experimental results show that it is possible to accurately recognize object instances by touch alone, including instances of novel objects that were never seen during training. Our learned model outperforms other methods on this complex task, including that of human volunteers.

title: Deep unsupervised network for multimodal perception, representation and classification
abstract: In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from th.

title: Autonomous dishwasher loading from cluttered trays using pre trained deep neural networks
abstract: Autonomous dishwasher loading is a benchmark problem in robotics that highlights the challenges of robotic perception, planning and manipulation in an unstructured environment. Current approaches resort to a specialized solution, however, these technologies are not viable in a domestic setting. Learning based solutions seem promising for a general purpose solutions, however, they require large amounts of catered data, to be applied in real world scenarios. This paper presents a novel solution based on pre trained object detection networks. By developing a perception, planning and manipulation framework around an off the shelf object detection network, we are able to develop robust pick and place solutions that are easy to develop and general purpose requiring only a RGB feedback and a pinch gripper. Analysis of a real world canteen tray data is first performed and used for developing our in lab experimental setup. Our results obtained from real world scenarios indicate that such approaches are highly desirable for plug and play domestic applications with limited calibration. All the associated data and code of this work is shared in a public repository.

title: Applied machine learning for multi sensory robot perception
abstract: In recent years, advances in autonomous robotics have begun to transform how we work and live. Unmanned Aerial Vehicles (UAV) and Unmanned Ground Vehicles are helping us to deliver goods, conduct surveys of construction sites, and perform search and rescue alongside first responders. However, designing robots with this level of autonomy is often challenging due to the complexity of the real world environment. Multi sensory perception is a critical component to address this challenge and develop robust autonomous robotic systems. By combining multiple inputs from sensors, the system can eliminate a single point of failure from sensor degradation and generate new insights to make better decisions integrating information from different sensor modalities. Recent breakthroughs in Machine Learning, especially the Deep Neural Network(DNN) based deep learning perception pipelines have been proven effective in a number of robot perception tasks. However, the significant computation cost for Deep Neural Networks is prohibiting their deployment on a robot system with limited power budget and real time performance requirement. It is important to bridge this gap by optimization to deploy state of the art machine learning models to a real world robot systems. This work investigates the viability to develop robust multi sensory robot perception systems enhanced by machine learning models in three different chapters. First, I explore the effectiveness of DNN perception pipelines in object detection and semantic segmentation tasks, then experiment on various model optimization techniques to enhance the efficiency of these perception models, achieving real time performance on robot system with a limited power budget. Then I elucidate the design and implementation of a thermal sensing robot system that performs sensor fusion of a thermal camera and an RGB Depth Camera to automatically track occupants in a building, measuring their forehead temperature, providing fine grain information for better decision making in intelligent Air Conditioning (AC) system.

title: A Survey of Deep RL and IL for Autonomous Driving Policy Learning
abstract: Autonomous driving (AD) agents generate driving policies based on online perception results, which are obtained at multiple levels of abstraction, e.g. behavior planning, motion planning and control. Driving policies are crucial to the realization of safe, efficient and harmonious driving behaviors, where AD agents still face substantial challenges in complex scenarios. Due to their successful application in fields such as robotics and video games, the use of deep reinforcement learning (DRL) and deep imitation learning (DIL) techniques to derive AD policies have witnessed vast research efforts in recent years. This paper is a comprehensive survey of this body of work, which is conducted at three levels: First, a taxonomy of the literature studies is constructed from the system perspective, among which five modes of integration of DRL/DIL models into an AD architecture are identified. Second, the formulations of DRL/DIL models for conducting specified AD tasks are comprehensively reviewed, where various designs on the model state and action spaces and the reinforcement learning rewards are covered. Finally, an in depth review is conducted on how the critical issues of AD applications regarding driving safety, interaction with other traffic participants and uncertainty of the environment are addressed by the DRL/DIL models. To the best of our knowledge, this is the first survey to focus on AD policy learning using DRL/DIL, which is addressed simultaneously from the system, task driven and problem driven perspectives. We share and discuss findings, which may lead to the investigation of various topics in the future.

title: Representation of the environment in autonomous driving applications
abstract: CHORALE is a robotic team led by Philippe Martinet that studies autonomous robotics systems (autonomous vehicles, mobile robots, UAV, and heterogeneous multi robot system) from both the perception and control point of view, interacting in dynamic and complex environments that are often unknown a priori. The research objective is to develop and study new paradigms and concepts allowing autonomous systems to acquire and share a compact representation of the environment (taking explicitly into account the constraints of the task to be done) and to act/ interact in a collaborative way in order to realize safely and efficiently the navigation task in environments shared with humans (respecting social behavior rules) The team members skills mainly concern modelling, perception, control, autonomous navigation and deep learning (semantization, depth estimation) CHORALE aims to build a world representation according with the task to be done by potentialy heterogeneous robots (in terms of sensors and dynamics) to combine Model based approaches with Deep Learning techniques in order to extract features, to monitor the task execution and the dynamic update world representation, and to analyze a current situation (to feed a decision system)

title: Aggressive Perception Aware Navigation Using Deep Optical Flow Dynamics and PixelMPC
abstract: Recently, vision based control has gained traction by leveraging the power of machine learning. In this work, we couple a model predictive control (MPC) framework to a visual pipeline. We introduce deep optical flow (DOF) dynamics, which is a combination of optical flow and robot dynamics. Using the DOF dynamics, MPC explicitly incorporates the predicted movement of relevant pixels into the planned trajectory of a robot. Our implementation of DOF is memory efficient, data efficient, and computationally cheap so that it can be computed in real time for use in an MPC framework. The suggested Pixel Model Predictive Control (PixelMPC) algorithm controls the robot to accomplish a high speed racing task while maintaining visibility of the important features (gates) This improves the reliability of vision based estimators for localization and can eventually lead to safe autonomous flight. The proposed algorithm is tested in a photorealistic simulation with a high speed drone racing task.

title: Towards Monocular Vision based Obstacle Avoidance through Deep Reinforcement Learning
abstract: Obstacle avoidance is a fundamental requirement for autonomous robots which operate in, and interact with, the real world. When perception is limited to monocular vision avoiding collision becomes significantly more challenging due to the lack of 3D information. Conventional path planners for obstacle avoidance require tuning a number of parameters and do not have the ability to directly benefit from large datasets and continuous use. In this paper, a dueling architecture based deep double Q network (D3QN) is proposed for obstacle avoidance, using only monocular RGB vision. Based on the dueling and double Q mechanisms, D3QN can efficiently learn how to avoid obstacles in a simulator even with very noisy depth information predicted from RGB image. Extensive experiments show that D3QN enables twofold acceleration on learning compared with a normal deep Q network and the models trained solely in virtual environments can be directly transferred to real robots, generalizing well to various new environments with previously unseen dynamic objects.

title: Learning features while learning to classify: a cognitive model for autonomous systems
abstract: We describe a computational cognitive model intended to be a generalizable classifier that can provide context based feedback to semantic perception in robotic applications. Many classifiers (including cognitive models of categorization) perform well at the task of associating features with objects. Underlying their performance is an effective selection of the features used during classification. This feature selection (FS) process is usually performed outside the boundaries of the models that learn and perform classification tasks, often by human experts. In contrast, the cognitive model we describe simultaneously learns which features to use, as it learns the associations between features and classes. This integration of FS and class learning in one model makes it complementary to other machine learning techniques that generate feature based representations (e.g. deep learning methods) But their integration in a cognitive architecture also provides a means for creating a dynamic context that includes disparate sources of information (e.g. environmental observations, task knowledge, commands from humans) This richer context, in turn, provides a means for making semantic perception goal directed. We demonstrate automated FS, integrated with an instance based learning approach to classification, in an ACT R model of categorization by labeling facial expressions of emotion (e.g. happy, sad) and then generalizing the model to the classification of indoor public spaces (e.g. cafes, classrooms)

title: Off Policy Reinforcement Learning for Autonomous Driving
abstract: Modern autonomous driving systems continue to face the challenges of handling complex and variable multi agent real world scenarios. Some subsystems, such as perception, use deep learning based approaches to leverage large amounts of data to generalize to novel scenes. Other subsystems, such as planning and control, still follow the classic cost based trajectory optimization approaches, and require high efforts to handle the long tail of rare events. Deep Reinforcement Learning (RL) has shown encouraging evidence in learning complex decision making tasks, spanning from strategic games to challenging robotics tasks. Further, the dense reward structure and modest time horizons make autonomous driving a favorable prospect for applying RL. As there are practical challenges in running RL online on vehicles and most self driving companies have millions of miles of collected data, it motivates the use of off policy RL algorithms to learn policies that can eventually work in the real world. We explore the use of an off policy RL algorithm, Deep Q Learning, to learn goal directed navigation in a simulated urban driving environment. Since Deep Q Learning methods are susceptible to instability and sub optimal convergence, we investigate different strategies to sample experiences from the replay buffer to mitigate these issues. We also explore combining expert agent's demonstration data with the RL agent's experiences to speed up the learning process. We demonstrate promising results on the CoRL2017 and NoCrash benchmarks on CARLA.