# Sem-SLAM: Semantic-Integrated SLAM Approach for 3D Reconstruction
## 1 Introduction
The combination of these two technologies not only enables real-time updates of the environmental model but also significantly enhances the SLAM system's understanding of the environment by providing semantic information such as object categories and locations [5]. In contrast to existing methods, most of which struggle with high computational costs, resource consumption, and real-time performance issues, this paper fully exploits the advantages of NeRF and SLAM technologies. By leveraging cross-attention and self-attention mechanisms for information extraction, we construct a 3D scene reconstruction system with better real-time performance. Unlike other methods that lack comprehensive information extraction, we propose an innovative multi-scale information extraction scheme. Using multi-layer perceptrons as fundamental units, we build a multi-scale feature extractor to obtain semantic, RGB, and Truncated Signed Distance Field (TSDF) values, ultimately realizing information decoding through rendering techniques. This enables a more detailed and accurate understanding of the environment. Existing NeRF-based SLAM approaches often have limitations in semantic feature fusion. In response, we implement a full-attention cross-modal feature fusion method, which effectively integrates geometric, shape, and semantic features. Moreover, many current methods do not use a well-designed loss function system. We employ color loss, appearance loss, feature loss, and depth loss to guide the optimization of the network, leading to better mapping accuracy and object recognition. Extensive experiments on the Replica and ScanNet datasets demonstrate that our method surpasses existing NeRF-based SLAM approaches in mapping quality while exhibiting exceptional real-time semantic mapping capabilities. To address the limitations of existing methods, we summarize the key contributions as follows:  
* We propose a NeRF-based visual semantic SLAM system. To accurately construct a semantic map for mapping, we introduce a multi-scale information extractor to extract both coarse-grained and fine-grained information from the camera pose. This enables a more comprehensive and detailed understanding of the spatial information related to the camera, laying a solid foundation for precise mapping. * We implement a full-attention cross-modal feature fusion method to integrate geometric, shape, and semantic features based on cross-attention and self-attention. This innovative structure can fully utilize the functions of features at various levels. By effectively combining different types of features, it enhances the system's ability to perceive and analyze the environment, resulting in more accurate and meaningful semantic representations. * Additionally, we employ color loss, appearance loss, feature loss, and depth loss to guide the optimization of the network. These carefully designed loss functions play a crucial role in adjusting the network parameters, helping the system to converge towards better solutions. As a result, we can obtain more optimal results in terms of mapping accuracy, object recognition, and overall system performance. * We conduct extensive evaluations on two challenging datasets, Replica [6] and ScanNet [7]. The experimental results clearly demonstrate that, compared with existing NeRF-based SLAM methods, our method exhibits state-of-the-art performance in mapping, tracking, and semantic segmentation. It not only shows higher accuracy in mapping the environment but also achieves more precise object tracking and semantic segmentation, highlighting the superiority and practical value of our proposed Sem-SLAM. The remainder of this paper is organized as follows: In Section 2, we review typical methods for NeRF and Semantic SLAM. In Section 3, we present our proposed method, including the forward feature fusion network and the backward semantic mapping network. Section 4 describes the experimental settings and results, including the datasets, evaluation metrics, and qualitative and quantitative comparisons with existing methods.