# Real-Time Hybrid Multi-Sensor Fusion Framework for Perception in Autonomous Vehicles
## 5 Experiments Procedure Overview  
For our FCNx architecture, we use a combined dataset for training. The dataset is a combination of UC Berkeley DeepDrive (BDD) dataset [41], University of Toronto KITTI dataset [42] and a self-generated dataset generated from our sensors installed in our test vehicle. We validated the proposed algorithm on a dataset with 3000 training and 3000 testing road scene samples. Our sensor data was acquired from a ZED camera and an Ouster LiDAR mounted on our test vehicle capturing data from streets of the city of Chicago. The BDD and KITTI dataset are annotated and labeled at object and pixel levels. For our dataset, we performed the annotating and labeling manually. We use these annotations as the ground truth labels. For our annotations, we mark the road with the color yellow.  
We trained the network by using the following tunable hyper-parameters. The parameter values are found based on trial and error with different values and combinations that give the best performance. Some of the hyper-parameters we picked are as follows: learning rate of \(2\times 10^{-4}\), batch size of 5 and the keep probability of 0.5. We train the network using an NVIDIA GeForce RTX2080 Ti Ubuntu 18.04.3 LTS machine with 64 GB memory; then upload the model to an embedded NVIDIA Xavier edge computer in our vehicle for real-time inference. From the obtained results of our network we calculate the Cross-Entropy (C.E.) loss. This metric gives a measure of how well our segmentation network is performing. We define our performance goals as C.E. loss, using validation and strive to improve upon other baseline benchmarks networks namely FCN8 [23] and U-Net [43]. Another important factor for automated driving is performing inference in real time. Our network experiment procedure is shown in Figure 12.  
There are different metrics to measure the performance of our free navigation space segmentation network like pixel accuracy, mean Intersect over Union (mIoU) and the Cross-Entropy (C.E.) loss. The most common loss function is a pixel-wise Cross-Entropy. In this loss, we compare each pixels predicted class to the ground truth image pixel labels. We repeat this process for all pixels in each image and take the average. In our segmentation we have two classes (i.e., binary), road and not-a-road. The Cross-Entropy loss is defined as:  
Figure 12: Overview of the experiment process. Training a Deep Neural Network model requires powerful computing resources; hence, for training we use a GPU powered workstation. Inference is less computationally expensive; thus, when the model is trained, we use an embedded GPU enabled device for testing and inference at the edge in-vehicle.  
\[C.E.Loss=-(p\cdot\log(\hat{p})+(1-p)\cdot\log(1-\hat{p})) \tag{16}\]  
where the probability of pixel ground-truth values \(y_{true}\) for the road defined as \(P(Y_{true}=road)=p\) and for not-a-road defined as \(P(Y_{true}=notroad)=1-p\). The predicted values \(y_{pred}\) for road and not-a-road are defined as \(P(Y_{pred}=road)=\hat{p}\) and \(P(Y_{pred}=notroad)=1-\hat{p}\).  
To show the implementation of single object tracking via the traditional EKF on our embedded edge computer, we use the LiDAR and radar sensor measurements (\(P_{x}\) and \(P_{y}\): the object's positions in x and y-direction; \(V_{x}\) and \(V_{y}\): the object's velocities in x and y-direction.) We use these measurement values in the Extended Kalman filter algorithm. For our testing purposes, the ground truth (the actual path of the object) is measured manually and used for calculating root mean square error (RMSE) from Equation (17) to measure single object tracking algorithm performance.  
\[R.M.S.E.=\sqrt{\frac{1}{n}\cdot\sum_{i=1}^{n}{||\hat{x}_{estimate}-\hat{x}_{ groundtruth}||_{2}^{2}}} \tag{17}\]  
where \(\hat{x}_{estimate}\) is the EKF predicted states and \(\hat{x}_{groundtruth}\) is the actual states of the test vehicle.