Query: visual tactile fusion for multimodal  robot perception

title: Deep interaction: Wearable robot assisted emotion communication for enhancing perception and expression ability of children with Autism Spectrum Disorders
abstract: Recent changes in both the social economy and people's living conditions and their habits have had an influence on the incidence of Autism Spectrum Disorder (ASD) which has brought huge economic and mental burden to the society and has become an urgent public health problem. The main symptom of autism in children is the presence of a social barrier, and one of the main reasons for that is a lack of emotional cognitive ability. However, the existing autism treatment systems intended for children pay little attention to the emotion cognition disorder. Besides, too much importance has been given to the interaction with children, while limiting the timeliness and movability of these systems. With the aim to address this shortcoming, in this work, we focus on the emotion cognition disorder and design a feasible solution for enhancing perception and expression ability of children with ASD. First, the first view emotional care system for children with ASD (First ECS) is developed using a wearable robot as a system carrier and realizing the deep emotional interaction with children with autism from the first view perspective. Emotion communication is used to meet high timeliness requirements for emotion transmission in the First ECS. Next, the emotional interaction mechanism that is applicable to the line of sight and non line of sight communication scenarios is introduced. Also, the emotion perception engine and emotion expression engine are designed. Subsequently, multimodal data collection and processing by a wearable affective robot are discussed. In addition, this paper introduces a multimodal data fusion method from the angle of emotion relevance and emotion computing model based on audio visual data. Finally, a demo platform is built to verify the feasibility of the proposed system.

title: Spatial Asymmetry in Tactile Sensor Skin Deformation Aids Perception of Edge Orientation During Haptic Exploration
abstract: Upper limb amputees rely primarily on visual feedback when using their prostheses to interact with others or objects in their environment. A constant reliance upon visual feedback can be mentally exhausting and does not suffice for many activities when line of sight is unavailable. Upper limb amputees could greatly benefit from the ability to perceive edges, one of the most salient features of 3D shape, through touch alone. We present an approach for estimating edge orientation with respect to an artificial fingertip through haptic exploration using a multimodal tactile sensor on a robot hand. Key parameters from the tactile signals for each of four exploratory procedures were used as inputs to a support vector regression model. Edge orientation angles ranging from 90 to 90 degrees were estimated with an 85 input model having an R 2 of 0.99 and RMS error of 5.08 degrees. Electrode impedance signals provided the most useful inputs by encoding spatially asymmetric skin deformation across the entire fingertip. Interestingly, sensor regions that were not in direct contact with the stimulus provided particularly useful information. Methods described here could pave the way for semi autonomous capabilities in prosthetic or robotic hands during haptic exploration, especially when visual feedback is unavailable.

title: Multimodal sensor based control for human robot interaction
abstract: The methods required to make humans and robots interact and collaborate are the subject of research in physical human robot interaction (pHRI) One of the main issues, particularly when physical contact can occur between the two, is safety. A second fundamental requirement to establish pHRI is intuitive control by the human operator, in particular when s/he is non expert. Sensor based control can address both issues, by closing the perception to action loop, in a way that may well be assimilated to that of the lower motor neurons in our nervous system. Traditionally, heterogeneous sensor data was fed to fusion algorithms (e.g. Kalman or Bayesian based) so as to provide state estimation for modeling the environment. However, since robot sensors generally measure different physical phenomena, it is preferable to use them directly in the low level servo controller rather than to apply them to multi sensory fusion or to design complex state machines. This idea, originally proposed in the hybrid position force control paradigm, when extended to multiple sensors brings new challenges to the control design; challenges related to the task representation and to the sensor characteristics (synchronization, hybrid control, task compatibility, etc. The rationale behind my work has precisely been to use sensor based control as a means to facilitate the physical interaction between robots and humans. In particular, I have utilized vision, proprioceptive force, touch and distance to address pHRI case studies, targeting different applications and robot platforms. My research has followed four main axes: teach and repeat navigation of wheeled mobile robots, collaborative industrial manipulation with safe physical interaction, force and visual control for interacting with humanoid robots, and shared robot control. Each of these axes will be presented here, before concluding with a general view of the issues at stake, and on the research projects that I plan to carry out in the upcoming years.

title: Gain field modulation mechanism in multimodal networks for spatial perception
abstract: Seeing is not just done through the eyes, it involves the integration of other modalities such as auditory, proprioceptive and tactile information, to locate objects, persons and also the limbs. We hypothesize that the neural mechanism of gain field modulation, which is found to process coordinate transform between modalities in the superior colliculus and in the parietal area, plays a key role to build such unified perceptual world. In experiments with a head neck eye's robot with a camera and microphones, we study how gain field modulation in neural networks can serve for transcribing one modality's reference frame into another one (e.g. audio signals into eyes' coordinate) It follows that each modality influences the estimations of the position of a stimulus (multimodal enhancement) This can be used in example for mapping sound signals into retina coordinates for audio visual speech perception.

title: Temporal Perception of Visual Haptic Events in Multimodal Telepresence System
abstract: Book synopsis: Haptic interfaces are divided into two main categories: force feedback and tactile. Force feedback interfaces are used to explore and modify remote/virtual objects in three physical dimensions in applications including computer aided design, computer assisted surgery, and computer aided assembly. Tactile interfaces deal with surface properties such as roughness, smoothness, and temperature. Haptic research is intrinsically multi disciplinary, incorporating computer science/engineering, control, robotics, psychophysics, and human motor control. By extending the scope of research in haptics, advances can be achieved in existing applications such as computer aided design (CAD) tele surgery, rehabilitation, scientific visualization, robot assisted surgery, authentication, and graphical user interfaces (GUI) to name a few. Advances in Haptics presents a number of recent contributions to the field of haptics. Authors from around the world present the results of their research on various issues in the field of haptics.

title: Bayesian and Neural Inference on LSTM Based Object Recognition From Tactile and Kinesthetic Information
abstract: Recent advances in the field of intelligent robotic manipulation pursue providing robotic hands with touch sensitivity. Haptic perception encompasses the sensing modalities encountered in the sense of touch (e.g. tactile and kinesthetic sensations) This letter focuses on multimodal object recognition and proposes analytical and data driven methodologies to fuse tactile and kinesthetic based classification results. The procedure is as follows: a three finger actuated gripper with an integrated high resolution tactile sensor performs squeeze and release Exploratory Procedures (EPs) The tactile images and kinesthetic information acquired using angular sensors on the finger joints constitute the time series datasets of interest. Each temporal dataset is fed to a Long Short term Memory (LSTM) Neural Network, which is trained to classify in hand objects. The LSTMs provide an estimation of the posterior probability of each object given the corresponding measurements, which after fusion allows to estimate the object through Bayesian and Neural inference approaches. An experiment with 36 classes is carried out to evaluate and compare the performance of the fused, tactile, and kinesthetic perception systems. The results show that the Bayesian based classifiers improves capabilities for object recognition and outperforms the Neural based approach.

title: Brain Inspired Perception, Motion and Control
abstract: In this chapter, a possible solution for the future real implementation of brain inspired perception (vision, audition and tactile) motion (the optimal path planning) and control (robots' behavior management) is further presented. Based on the results from Chaps. 2 5, a conceptual model is established to evaluate cognition efficiency of the vision brain, taking danger recognition as an example. Based on the vision hypothesis, the underwater robots with a deep vision system single shot multibox detector (SSD) can preliminarily link the robotic vision cognition module with the brain inspired perception, motion and control. Such a deep vision system can also be utilized to further enhance the performance of planetary exploration wheeled mobile robot in Chap. 5 or other robots. Core functional modules for future rebuilding a real vision brain, along with the major principles to implement a real brain cognition, are presented, which include memory, thinking, imagination, feeling, speaking and other aspects associated with visual perception. Realization of a vision brain not only includes the fusion of sensors, but also includes the fusion of feature and knowledge. Deep robotic vision is strongly suggested to be introduced into the future advanced robotic control system. At the end of this chapter, the intelligence extremes of the vision brain and the necessity for the avoidance of robots' threatening to human are theoretically analyzed, and therefore, the necessity to set an up limit for the development of artificial intelligence is theoretically demonstrated.

title: On Robustness of Multi Modal Fusion Robotics Perspective
abstract: The efficient multi modal fusion of data streams from different sensors is a crucial ability that a robotic perception system should exhibit to ensure robustness against disturbances. However, as the volume and dimensionality of sensory feedback increase it might be difficult to manually design a multimodal data fusion system that can handle heterogeneous data. Nowadays, multi modal machine learning is an emerging field with research focused mainly on analyzing vision and audio information. Although, from the robotics perspective, haptic sensations experienced from interaction with an environment are essential to successfully execute useful tasks. In our work, we compared four learning based multi modal fusion methods on three publicly available datasets containing haptic signals, images, and robots' poses. During tests, we considered three tasks involving such data, namely grasp outcome classification, texture recognition, and most challenging multi label classification of haptic adjectives based on haptic and visual data. Conducted experiments were focused not only on the verification of the performance of each method but mainly on their robustness against data degradation. We focused on this aspect of multi modal fusion, as it was rarely considered in the research papers, and such degradation of sensory feedback might occur during robot interaction with its environment. Additionally, we verified the usefulness of data augmentation to increase the robustness of the aforementioned data fusion methods.

title: Complementing visual tracking of moving targets by fusion of tactile sensing
abstract: Robot control in uncertain and dynamic environments can be greatly improved using sensor based control. Vision is a versatile low cost sensory modality, but low sample rate, high sensor delay and uncertain measurements limit its usability, especially in strongly dynamic environments. Vision can be used to estimate a 6 DOF pose of an object by model based pose estimation methods, but the estimate is typically not accurate along all degrees of freedom. Force is a complementary sensory modality allowing accurate measurements of local object shape when a tooltip is in contact with the object. In multimodal sensor fusion, several sensors measuring different modalities are combined together to give a more accurate estimate of the environment. As force and vision are fundamentally different sensory modalities not sharing a common representation, combining the information from these sensors is not straightforward. We show that the fusion of tactile and visual measurements enables to estimate the pose of a moving target at high rate and accuracy. Making assumptions of the object shape and carefully modeling the uncertainties of the sensors, the measurements can be fused together in an extended Kalman filter. Experimental results show greatly improved pose estimates with the proposed sensor fusion.

title: Towards multimodal haptics for teleoperation: Design of a tactile thermal display
abstract: Surgical robotics is among the most challenging applications of motion control. Present and future systems are essentially master slave systems. Our work focuses on force feedback and haptic interfaces. In this context, we study multimodal haptic interfaces, i.e. the fusion of force feedback, with other tactile information such as temperature or pressure. First results support the proposition that such multimodal haptic devices can help improve surgeon's dexterity and motion control. In order to strengthen this point, we investigate the psychophysics of thermal perception. This paper presents a device for temperature feedback that can be integrated in a multimodal haptic console. A finger sized tactile temperature display able to generate temperature gradients under the fingertip is presented along with first measurement results.