Query: multimodal perception in autonomous robots

title: Robust Fusion of LiDAR and Wide Angle Camera Data for Autonomous Mobile Robots
abstract: Autonomous robots that assist humans in day to day living tasks are becoming increasingly popular. Autonomous mobile robots operate by sensing and perceiving their surrounding environment to make accurate driving decisions. A combination of several different sensors such as LiDAR, radar, ultrasound sensors and cameras are utilized to sense the surrounding environment of autonomous vehicles. These heterogeneous sensors simultaneously capture various physical attributes of the environment. Such multimodality and redundancy of sensing need to be positively utilized for reliable and consistent perception of the environment through sensor data fusion. However, these multimodal sensor data streams are different from each other in many ways, such as temporal and spatial resolution, data format, and geometric alignment. For the subsequent perception algorithms to utilize the diversity offered by multimodal sensing, the data streams need to be spatially, geometrically and temporally aligned with each other. In this paper, we address the problem of fusing the outputs of a Light Detection and Ranging (LiDAR) scanner and a wide angle monocular image sensor for free space detection. The outputs of LiDAR scanner and the image sensor are of different spatial resolutions and need to be aligned with each other. A geometrical model is used to spatially align the two sensor outputs, followed by a Gaussian Process (GP) regression based resolution matching algorithm to interpolate the missing data with quantifiable uncertainty. The results indicate that the proposed sensor data fusion framework significantly aids the subsequent perception steps, as illustrated by the performance improvement of a uncertainty aware free space detection algorithm.

title: A Generalized Model for Multimodal Perception
abstract: In order for autonomous robots and humans to effectively collaborate on a task, robots need to be able to perceive their environments in a way that is accurate and consistent with their human teammates. To develop such cohesive perception, robots further need to be able to digest human teammates' descriptions of an environment to combine those with what they have perceived through computer vision systems. In this context, we develop a graphical model for fusing object recognition results using two different modalities computer vision and verbal descriptions. In this paper, we specifically focus on three types of verbal descriptions, namely, egocentric positions, relative positions using a landmark, and numeric constraints. We develop a Conditional Random Fields (CRF) based approach to fuse visual and verbal modalities where we model n ary relations (or descriptions) as factor functions. We hypothesize that human descriptions of an environment will improve robot's recognition if the information can be properly fused. To verify our hypothesis, we apply our model to the object recognition problem and evaluate our approach on NYU Depth V2 dataset and Visual Genome dataset. We report the results on sets of experiments demonstrating the significant advantage of multimodal perception, and discuss potential real world applications of our approach.

title: Deep unsupervised network for multimodal perception, representation and classification
abstract: In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from their multiple perceptual channels in an unsupervised way. Most of the approaches in the literature exploit engineered methods to process each perceptual modality. In contrast, robots should be able to acquire their own features from the raw sensors, leveraging the information elicited by interaction with their environment: learning from their sensorimotor experience would result in a more efficient strategy in a life long perspective. To this end, we propose an architecture based on deep networks, which is used by the humanoid robot iCub to learn a task from multiple perceptual modalities (proprioception, vision, audition) By structuring high dimensional, multimodal information into a set of distinct sub manifolds in a fully unsupervised way, it performs a substantial dimensionality reduction by providing both a symbolic representation of data and a fine discrimination between two similar stimuli. Moreover, the proposed network is able to exploit multimodal correlations to improve the representation of each modality alone. We propose a fully unsupervised algorithm for perception.The algorithm processes high dimensional, multimodal input.The output is a symbolic representation along with continuous traits.We apply it on a robotic task involving vision, proprioception and speech.

title: Combined 2D 3D categorization and classification for multimodal perception systems
abstract: In this article we describe an object perception system for autonomous robots performing everyday manipulation tasks in kitchen environments. The perception system gains its strengths by exploiting that the robots are to perform the same kinds of tasks with the same objects over and over again. It does so by learning the object representations necessary for the recognition and reconstruction in the context of pick and place tasks. The system employs a library of specialized perception routines that solve different, well defined perceptual sub tasks and can be combined into composite perceptual activities including the construction of an object model database, multimodal object classification, and object model reconstruction for grasping. We evaluate the effectiveness of our methods, and give examples of application scenarios using our personal robotic assistants acting in a human living environment.

title: Deep unsupervised network for multimodal perception, representation and classification
abstract: In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from th.

title: Organizing multimodal perception for autonomous learning and interactive systems
abstract: A stable perception of the environment is a crucial prerequisite for researching the learning of semantics from human robot interaction and also for the generation of behavior relying on the robots perception. In this paper, we propose several contributions to this research field. To organize visual perception the concept of proto objects is used for the representation of scene elements. These proto objects are created by several different sources and can be combined to provide the means for interactive autonomous behavior generation. They are also processed by several classifiers, extracting different visual properties. The robot learns to associate speech labels with these properties by using the outcome of the classifiers for online training of a speech recognition system. To ease the combination of visual and speech classifier outputs, a necessity for the online training and basis for future learning of semantics, a common representation for all classifier results is used. This uniform handling of multimodal information provides the necessary flexibility for further extension. We will show the feasibility of the proposed approach by interactive experiments with the humanoid robot ASIMO.

title: XDvision: Dense outdoor perception for autonomous vehicles
abstract: Robust perception is the cornerstone of safe and environmentally aware autonomous navigation systems. Autonomous robots are expected to recognise the objects in their surroundings under a wide range of challenging environmental conditions. This problem has been tackled by combining multiple sensor modalities that have complementary characteristics. This paper proposes an approach to multi sensor based robotic perception that leverages the rich and dense appearance information provided by camera sensors, and the range data provided by active sensors independently of how dense their measurements are. We introduce a framework we call XDvision where colour images are augmented with dense depth information obtained from sparser sensors such as lidars. We demonstrate the utility of our framework by comparing the performance of a standard CNN based image classifier fed with image data only with the performance of a two layer multimodal CNN trained using our augmented representation.

title: Deep learning robotic guidance for autonomous vascular access
abstract: Medical robots have demonstrated the ability to manipulate percutaneous instruments into soft tissue anatomy while working beyond the limits of human perception and dexterity. Robotic technologies further offer the promise of autonomy in carrying out critical tasks with minimal supervision when resources are limited. Here, we present a portable robotic device capable of introducing needles and catheters into deformable tissues such as blood vessels to draw blood or deliver fluids autonomously. Robotic cannulation is driven by predictions from a series of deep convolutional neural networks that encode spatiotemporal information from multimodal image sequences to guide real time servoing. We demonstrate, through imaging and robotic tracking studies in volunteers, the ability of the device to segment, classify, localize and track peripheral vessels in the presence of anatomical variability and motion. We then evaluate robotic performance in phantom and animal models of difficult vascular access and show that the device can improve success rates and procedure times compared to manual cannulations by trained operators, particularly in challenging physiological conditions. These results suggest the potential for autonomous systems to outperform humans on complex visuomotor tasks, and demonstrate a step in the translation of such capabilities into clinical use. Getting safe and fast access to blood vessels is vital to many methods of treatment and diagnosis in medicine. Robot assisted or even fully autonomous methods can potentially do the task more reliably than humans, especially when veins are hard to detect. In this work, a method is tested that uses deep learning to find blood vessels and track the movement of a patient's arm.

title: Combined 2 D 3 D Categorization and Classification for Multimodal Perception Systems November 17 2011
abstract: This article describes an object perception system for autonomous robots performing everyday manipulation tasks in kitchen environments. The perception system gains its strengths by exploiting that the robots are to perform the same kinds of tasks with the same objects over and over again. It does so by learning the object representations necessary for the recognition and reconstruction in the context of pick and place tasks. The system employs a library of specialized perception routines that solve different, well defined perceptual sub tasks and can be combined into composite perceptual activities including the construction of an object model database, multi modal object classification, and object model reconstruction for grasping. We evaluate the effectiveness of our methods, and give examples of application scenarios using our personal robotic assistants acting in a human living environment.

title: Combined 2D 3D categorization and classification for multimodal perception systems
abstract: In this article we describe an object perception system for autonomous robots performing everyday manipulation tasks in kitchen environments. The perception system gains its strengths by exploiting.