# Sem-SLAM: Semantic-Integrated SLAM Approach for 3D Reconstruction
## 3 Methodology
### Loss Function  
This section mainly introduces the loss functions involved in this paper, including feature loss, semantic loss, depth loss, and image loss. The design of the image loss and semantic loss mainly employs cross-entropy loss, L1 loss, and Structural Similarity (SSIM) loss, while the depth loss and feature loss only use the L1 norm.  
Specifically, our loss function consists of the following parts:  
For the image loss \(l_{rgb}\), we use the L1 loss and the Structural Similarity (SSIM) loss to calculate the difference between the rendered image and the real image. The L1 loss is used to calculate the absolute difference between the two images, and the SSIM loss is used to calculate the structural similarity between the two images. The calculation formula of the image loss is as follows:  
\[l_{rgb}=(1-\lambda_{rgb})\cdot\mathrm{L1}(I_{render},I_{gt})+\lambda_{rgb} \cdot\mathrm{SSIM}(I_{render},I_{gt}) \tag{22}\]  
Among them, the rendered image is \(I_{render}\), the ground truth image is \(I_{gt}\), and \(\lambda_{rgb}\) is the weight coefficient.  
For the semantic loss \(l_{sem}\), we also use the L1 loss and the SSIM loss to calculate the difference between the rendered semantic image and the ground truth semantic image. The calculation formula of the semantic loss is as follows:  
\[l_{sem}=(1-\lambda_{sem})\cdot\mathrm{L1}(S_{render},S_{gt})+\lambda_{sem} \cdot\mathrm{SSIM}(S_{render},S_{gt}) \tag{23}\]  
Among them, the rendered semantic image is \(S_{render}\), the ground truth semantic image is \(S_{gt}\), and \(\lambda_{sem}\) is the weight coefficient.  
For the depth loss \(l_{depth}\), we use the L1 loss to calculate the difference between the rendered depth image and the ground truth depth image. The calculation formula of the depth loss is as follows:  
\[l_{depth}=\mathrm{L1}(D_{render},D_{gt}) \tag{24}\]  
where \(D_{render}\) is the rendered depth image, and \(D_{gt}\) is the ground truth depth image.  
In order to enhance the perception ability of feature details, we adopt the feature loss to constrain the intermediate features in the rendering process and the feature extraction process. For the feature loss \(l_{feat}\), the L1 loss is typically used to calculate the difference between the rendered feature sequence and the feature sequence obtained from feature extraction. The calculation formula of the feature loss is as follows:  
\[l_{feat}=\mathrm{L1}(F_{render},F_{in}) \tag{25}\]  
where \(F_{render}\) is the rendered feature sequence, and \(F_{in}\) is the feature sequence obtained from feature extraction.  
The overall loss function of our model is defined as follows:  
\[L=\lambda_{r}l_{rgb}+\lambda_{s}l_{sem}+\lambda_{d}l_{depth}+\lambda_{f}l_{ feat} \tag{26}\]where \(\lambda_{r}\), \(\lambda_{s}\), \(\lambda_{d}\), and \(\lambda_{f}\) are the weight coefficients for the semantic loss, depth loss, and feature loss, respectively. The overall loss function combines the different components to guide the training process and improve the performance of the model.  
The selection of these loss weights (\(\lambda\) values) is based on extensive experiments. We conducted a series of experiments to explore different weight combinations and evaluated their effects on the model's performance using metrics such as mean Intersection over Union (mIoU) and Absolute Trajectory Error (ATE). Specifically, we performed a grid search on the weight space and selected the optimal combination that achieved the best performance on both the Replica and ScanNet datasets. The ablation experiments in Section 4 also provide strong evidence for the effectiveness of the chosen weights.