## 2 Robot Environment Perception for Navigation
### LiDAR and Camera Data Fusion
One of the drawbacks of the multi-level fusion method is the loss of small objects in the detection stage due to feature map down-sampling. Combining image and point cloud feature maps by RoI (Regions of Interest)-pooling decreases the fine-grained spatial details. MVX-Net [65] introduces a method to fuse point cloud and image data voxel-wise or point-wise. Two-dimensional CNNs are used for the image feature extraction process, and a VoxelNet [87] based network detects 3D objects in the voxel representation. The input 3D LiDAR point cloud is mapped to the 2D image for image feature extraction in the point-wise fusion method, and then voxelisation and processing are done using VoxelNet. In voxel-wise fusion, the point cloud is firstly voxelised and then projected onto the image-based 2D feature representation to extract features. This sequential approach achieved state-of-the-art performance for 3D object detection at the time of its publication. Object detection utilising one-stage models performs object proposal retrieval and bounding box prediction in a single process. These detection models are suitable for real-time autonomous robot decision-making scenarios. State-of-the-art single-stage object detection methods, such as [88], simultaneously process depth images and RGB images to fuse points with image features, and then the generated feature map is used for bounding box prediction. The introduced method [88] utilises, two CNN-based networks to parallelly process point cloud and RGB front-view images. One CNN identifies LiDAR features, and the other CNN extracts RGB image features. Then, these RGB image features are mapped into the LiDAR range view. Finally, mapped RGB and LiDAR image features are concatenated and fed into LaserNet [89] for object detection and semantic segmentation. This network has been trained in an end-to-end manner. The network training was done for 300K iterations with a batch size of 128, distributed over 32 GPUs. The image fusion, object detection, and semantic segmentation process took 38 milliseconds on an Nvidia Titan Xp GPU. Multimodal Semantic Segmentation  
Scene semantic segmentation assigns a semantic category label to each pixel in a scene image, and it can be regarded as a refinement of object detection [90]. A scene can incorporate obstacles, free space, and living creatures (not necessarily limited to these categories). The complete semantic segmentation of images applies these semantic categories to all pixels across an image. Many recent computer vision methods rely on CNN architectures. These networks favour dense depth image data over sparse data. In general, LiDAR sensors produce irregular, sparse depth measurements. Ref. [73] introduced a technique to utilise LiDAR sparse data and RGB images to achieve depth completion and semantic segmentation in a 2D view. This network can work with sparse depth measurement densities as low as 0.8%, and at the time of its publication, this method showed state-of-the-art performance on the KITTI benchmark dataset. The base network of this prior work was adopted from NASNet [91], which has an encoder and decoder architecture. Using LiDAR and RGB image feature fusion, ref. [92] proposed a novel method to achieve 2D semantic segmentation in 2D images. This method introduced a self-supervised network that suits different object categories, geometric locations, and environmental contexts. This self-supervised network uses two sensor modality-specific encoder streams, which concatenate to a single intermediate encoder and then connect into a decoder to fuse the complementary features. The segmentation part is achieved with a network termed AdapNet++ [92] that consists of an encoder-decoder architecture. All these network models were implemented using the deep learning TensorFlow library. Another high-performing deep learning-based LiDAR-Camera 2D semantic segmentation method was presented in [93].