Optimized Gated Deep Learning Architectures for Sensor Fusion

1 INTRODUCTION . Sensor fusion is an essential technology to autonomous systems such as self-driving cars and mobile robots . In advanced driver-assistance systems ( ADAS ) , many sensors such as cameras , ultrasonic sensors , and LiDARs are utilized for enhanced safety and driving experience . Sensor fusion based vehicle and robot control technologies have been explored ( Vargas-Meléndez et al. , 2016 ; Jain et al. , 2016 ; Garcia et al. , 2017 ; Bohez et al. , 2017 ; Patel et al. , 2017 ) . In addition , devices like smartphones and smartwatches typically integrate a number of sensors , making these devices a suitable platform for running sensor fusion applications such as activity recognition . Several sensor fusion techniques for activity recognition have been proposed ( Yurtman & Barshan , 2017 ; Dehzangi et al. , 2017 ; Zhao & Zhou , 2017 ; Gravina et al. , 2017 ) . More specifically , in Bohez et al . ( 2017 ) , a deep reinforcement learning based sensor fusion algorithm is discussed for robot control . A Kuka YouBot is used with multiple LiDAR sensors for simulations . In terms of sensor fusion , early fusion which concatenates all inputs as feature planes is compared with three other fusion techniques : concatenating convolution layer outputs , reducing the concatenated features with a 1x1 convolution , and accumulating convolution outputs . However , sensor noise and failures are not considered in this work . In addition , due to the fact that only the same type of sensory inputs is used in the experiments , the performance of sensor fusion based on different kinds of sensory inputs is unclear . Among sensor fusion techniques employed in automobiles , Vargas-Meléndez et al . ( 2016 ) exploit neural networks with a Kalman filter for vehicle roll angle estimation , and show the advantage of using an inertial measurement unit ( IMU ) without additional suspension deflection sensors . Jain et al . ( 2016 ) consider a sensor-rich platform with a learning algorithm for maneuver prediction . Long short-term memory ( LSTM ) networks , which are a type of recurrent neural networks ( RNN ) are used with sensory inputs from cameras , GPS , and speedometers . Garcia et al . ( 2017 ) propose joint probabilistic data fusion for road environments . However , neither a multiplicity of sensory inputs nor sensory noise and failures are considered . The adopted architecture is simplistic where input data are only fused in one layer . In the field of wearable devices , Yurtman & Barshan ( 2017 ) utilize early fusion , which concatenates sensory inputs . With this simple fusion approach , classical supervised learning methods such as Bayesian classifiers , k-nearest-neighbors , support vector machines , and artificial neural networks are compared . Dehzangi et al . ( 2017 ) use deep convolutional neural networks with IMU data . Zhao & Zhou ( 2017 ) use a CNN with angle embedded gate dynamic images , which are pre-processed inputs for gait recognition . Gravina et al . ( 2017 ) summarize three fusion methods , namely , data , feature , and decision level fusion . However , effective sensor fusion network architectures for coping with sensor failures are not deeply investigated . In terms of sensor fusion architectures , Patel et al . ( 2017 ) propose a so-called netgated architecture in which the information flow in a given convolutional neural network ( CNN ) is gated by fusion weights extracted from camera and LiDAR inputs . These fusion weights are used for computing a weighted sum of the sensory inputs . The weighted sum passes through fully connected layers to create a steering command . The gated networks ( netgated ) is shown to be robust to sensor failures , comparing to basic CNNs . However , a deep understanding of the relationships between sensory inputs , fusion weights , network architecture , and the resulting performances are not examined . The main objective of this paper is to propose optimized gated architectures that can address three limitations of the baseline netgated architecture of Patel et al . ( 2017 ) and investigate how different fusion architectures operate under clean sensory data and in the presence of snesor noise and failures . Our main contributions are : • Propose a new coarser-grained gated architecture which learns robustly a set of fusion weights at the ( feature ) group level ; • Further propose a two-stage gating architecture which exploits both the feature-level and group-level fusion weights , leading to further performance improvements . • Analyze the characteristics of the proposed architectures and show how they may address the limitations of the negated architecture in terms of inconsistency of fusion weights , overfitting , and lack of diverse fusion mechanisms . By utilizing driving mode prediction and human activity recognition datasets , we demonstrate the significant performance improvements brought by the proposed architectures over the conventional CNN and netgated architectures under various settings including cases where random sensor noise and failures are presented . Empirical evidence is also analyzed to help shed light on the underlying causes that may be responsible for the observed improvements of the two proposed architectures . 2 THE BASELINE NETGATED ARCHITECTURE AND ITS LIMITATIONS . 2.1 THE NETGATED ARCHITECTURE . The netgated architecture proposed in Patel et al . ( 2017 ) offers a promising approach for sensor fusion . This architecture was proposed under the context of unmanned ground vehicle ( UGV ) autonomous driving with convolutional neural networks with two sensors . A more general version of this architecture ( with five sensors/features ) is depicted in Fig . 1 . In Patel et al . ( 2017 ) , data from two sensory inputs , i.e . camera and LiDAR are processed individually through convolutional ( conv ) layers , pooling layers , and fully connected layers . The outputs from the fully connected ( FC ) layers ( e.g . “ FC-f1 ” to ” FC-f5 ” in the first dashed box in Fig . 1 ) , are concatenated and then fused by another FC layer ( e.g . “ FC-con ” in Fig . 1 ) , where two feature-level fusion weights are created . Feature-level fusion weights are originally referred to as scalars in Patel et al . ( 2017 ) . Note that each fusion weight is a scalar value and that in Fig . 1 five feature-level fusion weights are extracted which are the outputs of the “ FC-con ” layer . These fusion weights are multiplied with the corresponding outputs from the feature-level FC layers , i.e . the first dashed box , which is duplicated for better illustration of data flow in Fig . 1 . Finally , these weighted feature outputs are fused by the last FC layer ( i.e . “ FC-out ” ) , which produces the final prediction decision . The negated architecture is interesting in the sense that the extracted feature-level weights may be conceptually thought as a “ gated ” variable for each feature input and that a sensory input ( feature ) may be shut off from the network by the zero-valued fusion weight when it is corrupted by noise or sensor failures . As such , this architectures may be promising in providing robust sensor fusion capability . 2.2 LIMITATIONS OF THE BASIC NETGATED ARCHITECTURE . The negated architecture offers as an appealing end-to-end deep learning solution . Nevertheless , partially due to its end-to-end black-box nature , this architecture has several limitations as discussed below . Inconsistency of Fusion Weights . First , consider the situation in which there are N input features f1 , f2 , · · · , fN with the corresponding feature-level fusion weights fw1 , fw2 , · · · , fwN . As in Fig . 1 , the feature-level fusion weights are produced by the “ FC-con ” layer based on fused information from all inputs . As a result , an extracted fusion weight might not fully correspond to the corresponding feature due to the information sharing among all features . As we have observed in our experimental studies , there exist cases where the feature with the largest extracted fusion weight does not represent the most critical feature for the learning task . While the ranking of the featurelevel weights may reflect the relative importance of the corresponding features to a certain degree , the association between the two is not always consistent . In this paper , we refer to this phenomenon as inconsistency of fusion weights . It can be well expected that inconsistency of fusion weights may adversely affect the overall prediction accuracy , particularly when the fusion weights for certain noisy or corrupted features are not robustly learned by the network , resulting misleadingly large fusion weight values . Potential Over-fitting . Furthermore , for applications where many features need to be fused , using the same number fusion weight values introduces many additional parameters that shall be learned properly in the training process , making over-fitting of the model easier to occur . This situation further exacerbates due to the potential occurrence of inconsistency of fusion weights . Lack of Additional Fusion Mechanisms . Finally , in the architecture of Fig . 1 , apart from the learning of fusion weights , fusion of raw input features is done in a simplistic manner , i.e . by the last fully connected layer “ FC-out ” . Nevertheless , there exist more powerful raw input fusion mechanisms which could potentially lead to additional performance improvements . We address the above limitations of the baseline netgated architectures by proposing two extensions : a coarser-grained architecture and a hierarchical two-stage architecture , referred to as the FeatureGroup Gated Fusion Architecture ( FG-GFA ) and the Two-Stage Gated Fusion Architecture ( 2SGFA ) , respectively , described in the following sections . 3 FEATURE-GROUP GATED FUSION ARCHITECTURE ( FG-GFA ) . To address the aforementioned limitations of the baseline netgated architecture , we first explore the coarser-grained architecture , namely , Feature-Group Gated Fusion Architecture ( FG-GFA ) as in Fig . 2 , where for illustration purpose two feature groups are shown with the first group having three features and the second group two features . In general , a given set of N input features f1 , f2 , · · · , fN may be partitioned into , say M , feature groups FG1 , FG2 , · · · , FGM . As one specific example of this architecture , all features in a feature group are concatenated first , and then passed onto a convolution layer and a pooling layer . After going through the corresponding FC layer ( “ FC-g1 ” or “ FC-g2 ” in Fig . 2 ) , the processed information from all groups are concatenated and then passed onto an FC layer ( “ FC-con ” in Fig . 2 ) whose outputs are split into M group-level fusion weights . The fused information of each group e.g . ‘ FC-g1 ” or “ FC-g2 ” in Fig . 2 , is multiplied by the corresponding group-level fusion weight ( “ FC-g1 ” or “ FC-g2 ” are again duplicated in Fig . 2 to better illustrate the information flow ) . All weighted group-level information is combined and then processed by the final FC layer ( “ FC-out ” in Fig . 2 ) to produce the prediction decision . The configuration and the number of layers at each key step of the FG-GFA may be chosen differently from the specific example of Fig . 2 . We now comment on the key differences between the FG-GFA architecture and the baseline netgated architecture . First of all , in addition to the final fusion operation of the “ FC-out ” in Fig . 2 , we have performed additional early fusion of sensory inputs within each group . The outputs of such within-group fusions are combined to produce a smaller number of group-level fusion weights . Furthermore , the extracted group-level weights are used to multiply the corresponding fused group feature information , not the individual feature information . These characteristics of FG-GFA introduce different types of fusion mechanisms into the network . Second , since now fusion weights are extracted only at the group-level , fewer weights need to be learned compared with the baseline architecture . The fact that there are now a less number of tuning parameters and that early fusion takes place within each group might reduce the likelihood of stucking the training process on local minima . As a result , it may be possible to mitigate the issues of inconsistency of fusion weights and potential over-fitting . As will be demonstrated in the experimental study , FG-GFA leads to significantly more robust learning of fusion weights at the group level , i.e . existence of noisy or corrupted features in a group can be much more reliably reflected in the corresponding reduction of the grouplevel fusion weight . As a result , group-level fusion weights become a more reliable indicator of the quality of the sensory inputs in each group . Accordingly , we have empirically observed improved performance brought by FG-GFA as demonstrated later .

Summary: This paper tackles the problem of sensor fusion, where multiple (possibly differing) sensor modalities are available and neural network architectures are used to combine information from them to perform prediction tasks. The paper proposed modifications to a gated fusion network specifically: 1) Grouping sets of sensors and concatenating them before further processing, and 2) Performing multi-level fusion where early sensor data representations are concatenated to produce weightings additional to the those obtained from features concatenated at a later stage. Experimental results show that these architectures achieve performance gains from 2-6%, especially when sensors are noisy or missing.