# Semantic 3D Mapping from Deep Image Segmentation
## 3 Segmentation and Mapping in 3D
### 3D Segmentation  
3D segmentation from a segmented image and its corresponding point cloud is not simple. If we simply accumulate the points in the point cloud corresponding to each pixel in a mask, we will obtain a degraded result due to the _boundary point_ (Figure 3) problem. This problem arises when the depth image does not exactly match the color image where segmentation is performed, mainly due to errors or lack of accurate calibration. This problem is critical because it may place targets far from where they are. Figure 3: At the bottom left, we show the segmented image in RViz [[https://github.com/ros-visualization/rviz](https://github.com/ros-visualization/rviz)]([https://github.com/ros-visualization/rviz](https://github.com/ros-visualization/rviz)) (accessed on 21 January 2021). The rest of the figure shows the corresponding point cloud. The _Boundary point_ problem appears at the edges of the segmented areas. As seen in the point cloud, some points belong to the book whose positions are on the back wall. Figure 2: Complete process of our approach. The algorithm calculates an octomap with all the detections from a segmented image and its corresponding point cloud. Each detection in the octomap has a color that determines what kind of object it belongs to. Using color as the key to indicate the detected object has several advantages:  
* The color of each object class is determined by 2D segmentation software, and it is not repeated. * The octomap allows assigning a color to each voxel of the octomap, so that the detections are uniquely marked. * It allows to visualize the octomap in a simple and intuitive way. Our 3D segmentation algorithm, presented in Algorithm 1, works as it follows:  
1. **[line 2]**: The function _deep_2d_segment_ returns a set of detections in the image. Each detection is a 2D bounding box containing a mask. Each pixel is the mask corresponding to the detected object, marked with the object class color as shown in Figure 2.
2. **[line 3]**: The result of the algorithm will be stored in _detection_octomap_, a colored octomap with each voxel containing a color and a probability. 3. **[lines 4-7]**: This loop iterates over each one of the detections in the image: (a) **[line 5]**: This algorithm uses a KD-tree [30] that uses a fast neighbor search algorithm [31]. This data structure is initialized with a point cloud, and allows querying which points are in the vicinity of a reference point. By having the point cloud registered with the image (the \(i\)th pixel of the image corresponds to the \(i\)th point of the point cloud) and the pixels that belong to a detection, we can create point clouds that contain only the 3D points of the detection made in the image. (b) **[line 6]**: Processing begins with a 3D point that belongs to the detected object. This 3D point is the result of obtaining the position of the point cloud that corresponds to one of the detection pixels in the image. The detection center or centroid is valid for this purpose. (c) **[line 7]**: Start the recursive processing to ensure octomap connectivity. The recursive function \(EXPAND\_octomap\), also presented in Algorithm 1, is called until all points representing the detected object in the output octomap are connected. As arguments, it receives the 3D point to expand, the KD-tree with the detected points and the octomap to store the result. It works as it follows:  
1. **[lines 2,3]**: Neighbors of the input point are calculated using a search in a radius \(kr\). \(kr\) is the voxel size in the output octomap. 2. **[lines 4,5]**: If a neighbor is found, it is included into the output octomap. Note that the output octomap will contain points that were not in the original detection point cloud, but rather represent a continuous space occupied by this point cloud. 3. **[lines 6-9]**: This function will be called recursively with the expansion points that are around the input point, to expand the output octomap in all axes. 4.