# Object Semantic Grid Mapping with 2D LiDAR and RGB-D Camera for Domestic Robot Navigation
## 4 Back-End  
### Object Point Cloud  
An object point cloud is a set of 3D points with object semantic labels. We use two steps to generate the cloud--namely, point cloud generation and refinement. The specific contents of each step are introduced below. The pinhole camera model describes the process of projecting a 3D point in the camera coordinate system to a 2D pixel in the image coordinate system. Defining a point in the camera coordinate system is \(\mathbf{X}^{\text{c}}=[\ x^{c},y^{c},z^{c}]^{\text{T}}\); its corresponding pixel \(\mathbf{u}\) is:  
\[\mathbf{u}=\left[ \begin{array}{c}u\\ v\\ 1\end{array} \right]=\frac{1}{z^{\text{c}}}\left[ \begin{array}{ccc}f_{x}&0&c_ {x}\\ 0&f_{y}&c_{y}\\ 0&0&1\end{array} \right]\left[ \begin{array}{c}x^{\text{c}}\\ y^{\text{c}}\\ z^{\text{c}}\end{array} \right]=\frac{1}{z^{\text{c}}}\mathbf{K}\mathbf{X}^{ \text{c}}, \tag{4}\]  
where K is the camera intrinsic and can be obtained by calibration. As shown in Equation (4), depth information is lost during this projection process, resulting in the inability to recover the 3D point by a color image alone. However, an RGB-D camera can obtain both depth image and color image simultaneously. It can directly obtain \((u,v,z^{\text{c}})\). Therefore, based on the RGB-D sensor, the back-projection process of the pinhole camera can generate the 3D point in the camera coordinate system:  
\[\mathbf{X}^{\text{c}}=z^{\text{c}}\mathbf{K}^{-1}\mathbf{u}. \tag{5}\]  
We further use depth and object height constraints to improve the accuracy of the point cloud. The error of the depth image increases with the distance, and the interpolated camera poses may be not very accurate, so we only use the depth value smaller than a certain range. The object height constraint requires that the 3D object point is within a certain range. We use this constraint to reduce the error results from Mask R-CNN. For example, the model provided by Mask R-CNN is difficult to distinguish between a sink and a chandelier, but the chandelier hangs on the wall and the sink stands on the ground. According to the constraint of the sink, the error results can be removed. Finally, as shown in Figure 5, a 3D point cloud with semantic information can be generated. Due to sensor noise, inaccurate camera poses, and object detection errors, the point cloud contains many noise and error points. Moreover, the point cloud only includes object semantic labels but does not distinguish different instances. For example, although the environment contains two couches, the point cloud treats them as one. Therefore, the point cloud filter method is used to remove some noise points, and the point cloud segmentation method is employed to obtain different object instances. Noise points are relatively sparse, while the points within the object are more concentrated. According to this characteristic, we use the statistical filter to remove some noise points [41]. This method assumes that the average distance between a point and its k neighbors follows a Gaussian distribution, then calculates the mean and standard deviation of the distribution, and, finally, takes the points outside the standard deviation as noise points and removes them. To distinguish different objects with the same semantic label, the Euclidean cluster segmentation is adopted to generate object instances [42]. This method first randomly selects a point in the point cloud and simultaneously generates a new instance, then calculates the nearest neighbors that meetsthe Euclidean distance threshold and adds them to the instance. Repeat this process until no neighbors can be added to the instance to complete the instance segmentation. At last, repeat the aforementioned process for the remaining point clouds, until all points have an instance label. Finally, we use the instance minimum number constraint to further refine the object points. In other words, the object instances whose numbers do not satisfy this constraint are removed.