# Sem-SLAM: Semantic-Integrated SLAM Approach for 3D Reconstruction
## 3 Methodology
### Backward Semantic Mapping  
There are usually two common rendering methods in existing models. One approach [23] employs independent rendering networks to process different features. The other method [24] utilizes a decoder network to obtain geometric and color information from a single feature. However, neither of these two methods takes into account the information of different scales covered in the images. In our work, we integrate the multi-scale mesh feature extraction method into the rendering process and combine it with a multi-layer perceptron (MLP) to obtain Signed Distance Function (SDF), RGB, and semantic values from geometric, appearance, and semantic features. It has been found that the introduction of multi-scale mesh feature extraction improves the performance of implicit semantic modeling and provides a more meticulous and rich semantic understanding. The multi-scale extractor addresses the limitations of traditional NeRF-SLAM methods in handling multi-scale information in complex environments. Specifically, previous NeRF-SLAM methods were usually limited to single-scale information processing and performed poorly when dealing with large scenes or areas rich in details. In contrast, our multi-scale extractor significantly improves the accuracy of environmental under standing and reconstruction by simultaneously extracting coarse-grained and fine-grained information. For example, coarse-grained information helps to grasp the overall scene structure (e.g., room layout), while fine-grained information can capture local detailed features (e.g., object edges). This multi-scale processing capability enables the system to obtain more accurate environmental representations at various scales, thereby enhancing the quality of 3D reconstruction and the accuracy of semantic understanding. In this paper, we define multi-scale discrete feature meshes for each feature type (semantic, RGB, geometric). The meshes at each scale have different resolutions, and the corresponding features are extracted from the given spatial coordinates through trilinear interpolation. Meanwhile, we concatenate the multi-scale features under the same feature type to form a hybrid representation. This concatenation operation can combine features of different scales, thus enhancing the model's ability to capture both the macroscopic structure and local details. Eventually, we concatenate the semantic, RGB, and geometric features into a global feature vector. Finally, the global feature is input into the MLP network, which decodes it to generate semantic labels, RGB colors, and geometric densities, respectively. The semantic labels are used to represent the semantic information of the scene, the RGB colors are used to represent the appearance of the scene, and the geometric densities are used to represent the geometric structure of the scene. We then provide the following mathematical derivation. Let \(M^{k}\) denote the mesh at scale \(k\), where \(k\in\{1,\cdots,K\}\) and \(K\) is the total number of scales. For a given spatial coordinate \(p\), the feature \(f^{k}(p)\) at scale \(k\) can be obtained through trilinear interpolation:  
\[f^{k}(p)=\sum_{i=1}^{8}w_{i}^{k}(p)\cdot f_{v_{i}^{k}} \tag{7}\]  
where \(w_{i}^{k}(p)\) is the interpolation weight of the \(i\)-th vertex \(v_{i}^{k}\) of the voxel containing \(p\) at scale \(k\), and \(f_{v_{i}^{k}}\) is the feature value at vertex \(v_{i}^{k}\). The interpolation weights satisfy the following boundary conditions:  
\[\sum_{i=1}^{8}w_{i}^{k}(p)=1,\quad\text{for $p$ inside the voxel} \tag{8}\] \[w_{i}^{k}(p)\geq 0,\quad i=1,2,\cdots,8\] (9) \[w_{i}^{k}(p)=1\text{ if $p=v_{i}^{k}$,\quad$w_{j}^{k}(p)=0$ for $j\
eq i$} \tag{10}\]  
We choose trilinear interpolation in the multi-scale feature extractor for several reasons. Here, we provide a comparison with alternative interpolation methods to justify this choice.