# Semantic 3D Mapping from Deep Image Segmentation
## 3 Segmentation and Mapping in 3D
### Semantic Mapping  
The semantic mapping adds spatial occupation information of the detected elements to the environment map using the previous section's algorithm. The starting map is a 2D occupation grid where each cell represents the probability of an obstacle. Figure 5 shows where the obstacles are in the dark areas, the clear region is white, and the gray area is unknown. Each axis of reference is named, and we will call it a frame, e.g., the frame where the map is located is the _frame map_. More frames are related to each other by an invertible transformation matrix \(RT_{4\times 4}\) representing a translation and a rotation. The frames form a tree of frames, where each node is a frame, and the arcs are geometric transformations of translation and rotation. There can be no loops in this tree, and each frame has only one predecessor in the tree. In a robot, for example, there is one for each sensor or joint. The root of a robot's structure is called by convention _base_footprint_ (Figure 6), and it is at ground level, in the center of the robot. Any 3D coordinate in any frame can be transformed to coordinates in another frame. For example, the coordinate of an obstacle perceived in the _laser frame_ can be transformed into a coordinate of the _base_footprint frame_. The root of a robot's frame tree is the _odom frame_, whose child is _base_footprint frame_. The arc that joins them indicates the perceived displacement since the robot was started. This odometry is updated based on the robot's proprioceptive sensors. Figure 5: Grid map of the simulated environment. Figure 6: Coordinate axes connections from the map frame to the sensor frame. In this work, the robot is located using a particle filter. The results of this localization process are:  
* The connection of the frame map with the root of the robot tree, the frame odom: \(RT^{map\to odom}\). * A covariance matrix \(E_{6\times 6}\) that indicates the precision of the robot's location \((x,y,z,roll,pitch,yaw)\) in space. \(\mathcal{O}_{t}\) is the octomap that contains the cells that correspond to the objects detected in the 3D segmentation process described in the previous section. Each cell is a voxel in the space with an object class (coded by its color) and a probability. In the semantic mapping, we maintain a semantic map as an octomap \(\mathcal{M}_{t}\) in _frame map_, that updates with each instant perception \(\mathcal{O}_{t}\) (Figure 7). Perceptions \(\mathcal{O}_{t}\) are located in the frame of the sensor with which they were captured. If a robot is located, it is possible to obtain each cell of the octomap in the frame map, since there is a path in the frame tree from the sensor frame to the frame map, as shown in Equation (1). \[RT^{map\to sensor}=RT^{map\to odom\to base\_ footprint\to base\_link\to sensor}=RT^{map\to odom}*RT^{ odom\to base\_ footprint\to base\_link}*RT^{base\_link\to sensor}\] \[p^{map}=(RT^{map\to sensor})^{-1}*p^{sensor},\forall p\in \mathcal{O}_{t} \tag{1}\]  
We can not register \(p^{map}\) directly in \(\mathcal{M}_{t}\) because:  
* There is an uncertainty in \(RT^{map\to sensor}\) since the particle filter estimates the robot's position with uncertainty represented by the covariance matrix \(\mathbf{E}\) of the location. * There may be erroneous perceptions sporadically due to some error in the detection of the objects in the image, or in some synchronization between the point cloud and the image. For this reason, we have developed an algorithm that incrementally composes new octomaps taking into account the uncertainty in the location of the robot. The application steps for each new \(\mathcal{O}_{t}\) are as follows:  
Figure 7: Instantaneous perception \(\mathcal{O}_{t}\) from the 3D segmentation process. 1. Obtain the current uncertainty E of the location. \[\text{E}=\left(