# GSV-NET: A Multi-Modal Deep Learning Network for 3D Point Cloud Classification
## 1 Introduction  
3D sensors, including various LiDAR types, 3D scanners, and RGB-D cameras, such as RealSense, Kinect, and Apple depth cameras, have become more affordable and available with the fast evolution of 3D acquisition technologies [1]. The 3D data collected by these sensors can give various types of scaling geometric shape information [2, 3, 4]. The 3D data have more information than 2D images and allow machines to understand their surrounding environment better. Therefore, they have various applications in many fields, including robotics, self-driving cars, medical treatment, and remote sensing [5]. Different formats, including point cloud, depth images, volumetric grids, and meshes, can represent 3D data. Point cloud representation is a popular format that keeps the native geometric description in 3D space. Point clouds are not only obtained by LiDAR sensors, depth cameras, stereo cameras, etc., but are also enhanced by additional sensors, such as multispectral, thermal, or color information [6, 7, 8]. The 3D point cloud stores 3D coordinates (x, y, z) and adds properties such as intensity and reflection. Furthermore, RGB-D images can construct the point cloud, and every (x, y) coordinate will match with four properties (R, G, B, and depth). Point cloud classification can be applied to scene understanding such as robotics and autonomous driving [9; 10; 11]. In addition, point cloud classification is an essential task for the autonomous driving car because it helps the self-driving car identify objects on the roads. Some studies introduce several machine learning-based methods for point cloud classification [12; 13; 14]. Deep learning approaches have dominated various recent research fields, such as speech recognition or computer vision. Deep learning cannot apply directly to point cloud due to their irregular structure. Several researchers propose various deep learning-based solutions for point cloud classification, based on the view, the voxel, the raw-point cloud, and the graph. However, these approaches suffer from one or both of the following weaknesses: (1) missing test results with the real-world data that take from the LiDAR sensor, (2) having insufficiently good performance. In this work, we introduce a novel approach to overcome these above limitations. Most classification techniques in the 3D point cloud cannot catch both visual and structural data because they employ a singular modal (see reference [15]). As a result, the classification rate is not high. We can obtain the advantage of each feature description from various modal information to improve classification accuracy. Multi-modals have potential development in point cloud classification. However, creating effective multi-modal is a challenging task. In this paper, we propose a novel framework GSV-NET, that employs both the point cloud and view-based model. GSV-NET captures both the global and the regional point cloud features with Gaussian Supervector (GSV) and the enhancing region representation (ERR). The complement of these features enriches the point cloud description and boosts the classification rate. Furthermore, combining Gaussian Supervector and enhancing region representation helps create an effective multi-modal. Figure 1 shows the structures of the proposed framework. In this paper, our contributions are as follows:  
* We present a novel approach to extract the global point cloud feature using GSV representation and the 3D wide-inception architecture. * After converting 3D point cloud regions into color representation, a 2D wide-inception network is employed to extract the regional feature of the 3D point cloud. In addition, we present the 1D convolution neural network (CNN) structure that fuses the extracted global and regional features. The GSV and ERR are novel to the best knowledge of the authors. * Based on our numerical outcomes on challenging databases, the proposed approach is more accurate and efficient than the well-known methods. The paper structure is as follows.