# Sem-SLAM: Semantic-Integrated SLAM Approach for 3D Reconstruction
## 3 Methodology
### Backward Semantic Mapping
**Comparison with Nearest-Neighbor Interpolation**: Nearest-neighbor interpolation is the simplest method, which assigns the value of the nearest vertex to the interpolation point. While it is computationally efficient, it often results in a blocky or pixelated appearance, especially when the feature mesh has a low resolution. This is because it does not consider the contribution of neighboring vertices, leading to a lack of smoothness in the interpolated features. In contrast, trilinear interpolation uses the values of all eight vertices of the voxel, which can generate smoother and more accurate feature values, thus better preserving the details of the multi-scale features. **Comparison with Bilinear Interpolation**: Bilinear interpolation is commonly used in 2D scenarios. It considers the contribution of four neighboring points in a 2D plane. However, in our 3D multi-scale feature extraction task, the spatial information is three-dimensional. Bilinear interpolation cannot fully utilize the 3D spatial information, which may lead to information loss in the depth dimension. Trilinear interpolation, on the other hand, extends the concept of bilinear interpolation to 3D space, taking into account the contributions of all vertices in the 3D voxel. This allows it to better adapt to the 3D nature of our multi-scale features and capture the spatial relationships more comprehensively. For a specific feature type \(t\) (e.g., semantic, RGB, geometric), the hybrid feature \(h_{t}(p)\) is formed by concatenating multi-scale features:  
\[h_{t}(p)=[f_{t}^{1}(p);\cdots;f_{t}^{K}(p)] \tag{11}\]  
Then, we concatenate semantic, RGB, and geometric hybrid features into a global feature vector \(g(p)\):  
\[g(p)=[h_{sem}(p);h_{rgb}(p);h_{geo}(p)] \tag{12}\]  
The MLP network \(\mathcal{N}_{h}eta\) decodes the global feature to generate semantic labels \(\text{sem}(p)\), RGB colors \(\text{rgb}(p)\), and geometric densities \(\sigma_{geo}(p)\):  
\[\text{sem}(p) =\mathcal{N}_{sem}(g(p)) \tag{13}\] \[\text{rgb}(p) =\mathcal{N}_{rgb}(g(p))\] (14) \[\sigma_{geo}(p) =\mathcal{N}_{geo}(g(p)) \tag{15}\]  
To integrate these multi-scale mesh features with the TSDF representation, we use the geometric density \(\sigma_{geo}(p)\) to update the TSDF value \(d(p)\)[25]. The TSDF value represents the signed distance from the point \(p\) to the nearest surface, and we can use the following relationship to update it:  
\[d(p)=\alpha_{geo}\cdot\text{Sigmoid}^{-1}(\alpha_{geo}\cdot\sigma_{geo}(p)) \tag{16}\]  
where \(\alpha_{geo}\) is the learnable parameter for geometric rendering, controlling the sharpness of surface boundaries (suggested initialization range: \(\alpha_{geo}\in[1,10]\)). Among them, the rendering method references the approach of styleSDF. Along the ray \(r(t)=o+td\), \(N\) points \(p_{n}^{N}=\) are sampled. The multi-scale feature network \(D_{\theta}(p_{n})\) is used to generate the RGB value \(rgb(p_{n})\), the semantic value \(sem(p_{n})\), and the Truncated Signed Distance Field (TSDF) value \(d(p_{n})\) for each point. Subsequently, the Signed Distance Function (SDF) value is converted into the volume density:  
\[\sum_{i=1}^{8}w_{i}^{k}(p)=1,\quad\text{for $p$ inside the voxel} \tag{8}\]0 \[\sum_{i=1}^{8}w_{i}^{k}(p)=1,\quad\text{for $p$ inside the voxel} \tag{8}\]1  
where \(\alpha_{geo}\) and \(\alpha_{sem}\) are learnable parameters for geometric and semantic rendering, respectively, controlling the sharpness of surface boundaries and semantic distributions. The geometric volume density \(\sigma_{geo}(p_{n})\) is used to render color and depth:  
\[\sum_{i=1}^{8}w_{i}^{k}(p)=1,\quad\text{for $p$ inside the voxel} \tag{8}\]2  
\[\sum_{i=1}^{8}w_{i}^{k}(p)=1,\quad\text{for $p$ inside the voxel} \tag{8}\]3where \(d_{n}\) denotes the depth of point \(p_{n}\) relative to the camera pose. The semantic volume density \(\sigma_{sem}(p_{n})\) is used for semantic rendering:  
\[\sum_{i=1}^{8}w_{i}^{k}(p)=1,\quad\text{for $p$ inside the voxel} \tag{8}\]4