Query: deep multimodal data fusion for autonomous robot perception

title: Multimodal Deep Learning for Object Recognition Combining Camera and LIDAR Data
abstract: Object detection and recognition is a key component of autonomous robotic vehicles, as evidenced by the continuous efforts made by the robotic community on areas related to object detection and sensory perception systems. This paper presents a study on multisensor (camera and LIDAR) late fusion strategies for object recognition. In this work, LIDAR data is processed as 3D points and also by means of a 2D representation in the form of depth map (DM) which is obtained by projecting the LIDAR 3D point cloud into a 2D image plane followed by an upsampling strategy which generates a high resolution 2D range view. A CNN network (Inception V3) is used as classification method on the RGB images, and on the DMs (LIDAR modality) A 3D network (the PointNet) which directly performs classification on the 3D point clouds, is also considered in the experiments. One of the motivations of this work consists of incorporating the distance to the objects, as measured by the LIDAR, as a relevant cue to improve the classification performance. A new range based average weighting strategy is proposed, which considers the relationship between the deep models' performance and the distance of objects. A classification dataset, based on the KITTI database, is used to evaluate the deep models, and to support the experimental part. We report extensive results in terms of single modality i.e. using RGB and LIDAR models individually, and late fusion multimodality approaches.

title: Autonomous Navigation in Complex Environments with Deep Multimodal Fusion Network
abstract: Autonomous navigation in complex environments is a crucial task in time sensitive scenarios such as disaster response or search and rescue. However, complex environments pose significant challenges for autonomous platforms to navigate due to their challenging properties: constrained narrow passages, unstable pathway with debris and obstacles, or irregular geological structures and poor lighting conditions. In this work, we propose a multimodal fusion approach to address the problem of autonomous navigation in complex environments such as collapsed cites, or natural caves. We first simulate the complex environments in a physics based simulation engine and collect a large scale dataset for training. We then propose a Navigation Multimodal Fusion Network (NMFNet) which has three branches to effectively handle three visual modalities: laser, RGB images, and point cloud data. The extensively experimental results show that our NMFNet outperforms recent state of the art by a fair margin while achieving real time performance. We further show that the use of multiple modalities is essential for autonomous navigation in complex environments. Finally, we successfully deploy our network to both simulated and real mobile robots.

title: OLIMP: A Heterogeneous Multimodal Dataset for Advanced Environment Perception
abstract: A reliable environment perception is a crucial task for autonomous driving, especially in dense traffic areas. Recent improvements and breakthroughs in scene understanding for intelligent transportation systems are mainly based on deep learning and the fusion of different modalities. In this context, we introduce OLIMP: A heterOgeneous Multimodal Dataset for Advanced EnvIronMent Perception. This is the first public, multimodal and synchronized dataset that includes UWB radar data, acoustic data, narrow band radar data and images. OLIMP comprises 407 scenes and 47,354 synchronized frames, presenting four categories: pedestrian, cyclist, car and tram. The dataset includes various challenges related to dense urban traffic such as cluttered environment and different weather conditions. To demonstrate the usefulness of the introduced dataset, we propose a fusion framework that combines the four modalities for multi object detection. The obtained results are promising and spur for future research.

title: Multimodal visual image processing of mobile robot in unstructured environment based on semi supervised multimodal deep network
abstract: With the continuous development of computer technology, machine vision and image processing algorithms, people's research on mobile robots with vision systems is becoming deeper and deeper. This paper studies the related problems of visual image processing of mobile robots in outdoor unstructured environments. In this work, we propose a new approach that integrates heterogeneous features through a well designed Semi supervised multimodal deep network (SMMDN) For each modality, there is a multi layer sub neural network with a separate structure corresponding to it, which is used to transform features in different modes into the same modal features. At the same time, through a network layer common to all modes above these sub neural networks, a connection is established between these different modes, and finally a plurality of heterogeneous modes is converted into the same mode and a plurality of them are extracted from fusion characteristics of data modalities. The simulation results prove that SMMDN improves the perception and recognition ability of mobile robots for outdoor complex environments.

title: RGB D Object Classification for Autonomous Driving Perception
abstract: Autonomous driving systems (ADS) comprise, essentially, sensory perception (including AI ML based techniques) localization, decision making, and control. The cornerstone of an ADS is the sensory perception part, which is involved in most of the essential and necessary tasks for safe driving such as sensor fusion, environment representation, scene understanding, semantic segmentation, object detection/recognition, and tracking. Multimodal sensor fusion is an established strategy to enhance safety and robustness of perception systems in autonomous driving. In this work, a fusion of data from color camera (RGB) and 3D LIDAR (D distance) henceforth designated RGB D, will be particularly addressed, highlighting use cases on road users classification using deep learning. 3D LIDAR data, in the form of point cloud, can be processed directly by using the PointNet network or, alternatively, by using depth maps, known as range view representation, which is a suitable representation to train state of the art Convolutional Neural Network (CNN) models and to make the combination with RGB images more practical. Experiments are carried out using the KITTI dataset on object classification, i.e. vehicles, pedestrians, cyclists. We report extensive results in terms of classification performance of deep learning models using RGB, 3D, and RGB D representations. The results show that RGB D models have better performance in comparison with 3D and range view models but, in some circumstances RGB only achieved superior performance.

title: Multimodal sensory fusion for soccer robot self localization based on long short term memory recurrent neural network
abstract: Self localization is a fundamental requirement for autonomous mobile robots. With the rapid development in sensor technology, the sensor suites of robot provide multimodal information that naturally ensures perception robustness, multimodal sensory fusion are able to provide a better solution for enhance the capability of self localization. This paper proposes a multimodal sensory fusion method based on Long Short Term Memory (LSTM) Recurrent Neural Network (RNN) for RoboCup 3D Simulation league. This approach fuses Inertia Navigation System (INS) and vision perceptor information from different sensors at feature level instead of raw data. The experiment results demonstrate that the proposed approach makes an improvement in predictive accuracy and efficiency compared with the standard Extended Kalman Filter (EKF) and the static Particle Filter (PF) methods.

title: Research on autonomous driving perception based on deep learning algorithm
abstract: Multiple source sensor fusion is the foundation of motion planning for autonomous driving system, which is the crucial part in improving the performances for unmanned operational system. In this article, based on the deep learning platform CATARC constructed, applied with Udacity's Lincoln MKZ multiple sensor data, implemented with Robotic Operation System, Computer Vision, PointCloud Library, Deep Neural Networks and Extended Kalman Filter, constructed a low cost object pose estimation data fusion solution, aiming at technic support for the industrialization of autonomous driving technologies.

title: Robust Fusion of LiDAR and Wide Angle Camera Data for Autonomous Mobile Robots
abstract: Autonomous robots that assist humans in day to day living tasks are becoming increasingly popular. Autonomous mobile robots operate by sensing and perceiving their surrounding environment to make accurate driving decisions. A combination of several different sensors such as LiDAR, radar, ultrasound sensors and cameras are utilized to sense the surrounding environment of autonomous vehicles. These heterogeneous sensors simultaneously capture various physical attributes of the environment. Such multimodality and redundancy of sensing need to be positively utilized for reliable and consistent perception of the environment through sensor data fusion. However, these multimodal sensor data streams are different from each other in many ways, such as temporal and spatial resolution, data format, and geometric alignment. For the subsequent perception algorithms to utilize the diversity offered by multimodal sensing, the data streams need to be spatially, geometrically and temporally aligned with each other. In this paper, we address the problem of fusing the outputs of a Light Detection and Ranging (LiDAR) scanner and a wide angle monocular image sensor for free space detection. The outputs of LiDAR scanner and the image sensor are of different spatial resolutions and need to be aligned with each other. A geometrical model is used to spatially align the two sensor outputs, followed by a Gaussian Process (GP) regression based resolution matching algorithm to interpolate the missing data with quantifiable uncertainty. The results indicate that the proposed sensor data fusion framework significantly aids the subsequent perception steps, as illustrated by the performance improvement of a uncertainty aware free space detection algorithm.

title: Fuzzy multisensor fusion for autonomous proactive robot perception
abstract: Robot perception still lacks reliability in complex natural environments. A commonly used method to improve perception is to incorporate more sensors with different modalities. This leads to increased computational requirements due to the parallel processing of huge amounts of sensor data. Appropriate sensor fusion methods are needed if contradictory information is provided by different sensors. We propose a feature based technique to fuse multimodal sensor data using fuzzy rules. Probabilistic methods are avoided by applying fuzzyfication at the feature level. We propose a higher information gain of the available sensors by utilizing robot actions to focus sensors on objects of interest. Therefore sensor readings, algorithms and robot actions are combined into feature detectors. A goal directed activation of these feature detectors renders parallel processing of all sensor data unnecessary.

title: Unsupervised Machine Learning Based Scalable Fusion for Active Perception
abstract: The fusion of sensor data is an essential requirement for active perception in autonomous robots in order to accurately perceive an environment, make appropriate adjustments that improve the understanding of the environment and generate actionable insights. Despite a large body of literature in this domain, only a handful of research focuses on unsupervised machine learning, which is increasingly important in unlabeled, unknown environments. This paper proposes a scalable self organizing neural architecture for environmental perception based on multimodal fusion using unsupervised machine learning. Inspired by the biological counterpart, the neural architecture consists of topographic maps across each modality and incorporates a novel scalable self organization process that handles high volume, high velocity sensor data. The co occurrence relationships across modalities are captured in cross modal connections and the reverse Hebbian projection is used for multimodal representation. This proposed unsupervised machine learning based scalable fusion technique was implemented on Apache Hadoop and Apache Spark. It was validated using an extensive multimodal human activity sensor dataset to demonstrate efficient representation and multimodal fusion for active perception. Note to Practitioners Heterogeneous sensors are increasingly used to capture different aspects of an event or an environment. This paper presents a multimodal fusion mechanism to fuse such heterogeneous sensor data to support the accurate perception of the surrounding. The proposed mechanism uses natural regularities in patterns observed across multiple modalities and enriches each modality with information from other co occurring modalities to achieve a multimodal clustering. The cluster can be used for online prediction and offline action planning of the robot by matching the current sensory inputs. The algorithm uses unsupervised machine learning, and hence, can be used with any multimodal dataset without being restricted by the labeling of the dataset. In addition, we adapt the algorithm for multiple big data platforms so that it is effective and pragmatic for large scale sensor data fusion tasks.