Title: Understanding while Exploring: Semantics-driven Active Mapping
Venue: NeurIPS 2025
Accepted: yes
Keywords: Active Semantic Mapping

Active Mapping

Gaussian Splatting

Exploration
Abstract: Effective robotic autonomy in unknown environments demands proactive exploration and precise understanding of both geometry and semantics. In this paper, we propose ActiveSGM, an active semantic mapping framework designed to predict the informativeness of potential observations before execution. Built upon a 3D Gaussian Splatting (3DGS) mapping backbone, our approach employs semantic and geometric uncertainty quantification, coupled with a sparse semantic representation, to guide exploration. By enabling robots to strategically select the most beneficial viewpoints, ActiveSGM efficiently enhances mapping completeness, accuracy, and robustness to noisy semantic data, ultimately supporting more adaptive scene exploration. Our experiments on the Replica and Matterport3D datasets highlight the effectiveness of ActiveSGM in active semantic mapping tasks.


Peer reviews:
comments: SUMMARY:
The paper introduces ActiveSGM (Active Semantic Gaussian Mapping), a novel framework for active semantic mapping. This system combines semantic and geometric uncertainty quantification, guiding robots to explore their environments efficiently by strategically selecting the most informative viewpoints. It utilizes a 3D Gaussian Splatting (3DGS) backbone to generate dense, high-fidelity semantic maps, enhancing exploration and scene understanding. The framework has been tested on the Replica and Matterport3D datasets, showcasing superior performance in improving mapping completeness, accuracy, and robustness to noisy semantic data.

STRENGTHS AND WEAKNESSES:
Strength
1. The combination of active exploration with semantic and geometric uncertainty provides a meaningful advancement in robotic mapping. 
2. The paper demonstrates its effectiveness through comprehensive experiments on popular datasets like Replica and MP3D. 
3. 
Weakness
1. The use of pseudo labels from the pre-trained OneFormer model for semantic predictions introduces a dependency on external, potentially error-prone models. This could lead to inaccuracies, especially in complex or occluded scenes where the pretrained model might perform suboptimally.
2. The method may be limited by the initial semantic map’s accuracy, as semantic initialization from first observations can be unreliable, especially in occluded or unseen regions.

QUESTIONS:
See Weakness

LIMITATIONS:
See Weakness
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present the first method for active semantic mapping with neural radiance fields. They propose a representation in which cropped semantic probability vectors are stored in a 3DGS backbone. The objective for active mapping simultaneously incentivizes regions of high semantic entropy computed from the semantic probability vectors and low geometric coverage computed from missing pixel counts. This approach is validated on the Replica and Matterport3D datasets and shows additional capability over alternate approaches while simultaneously achieving strong (often state of the art) performance on accurate and efficient mapping.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this is a strong paper which I enjoyed reading. The method fills a clear gap in current literature and is a novel and well presented approach. The literature review is thorough and educational across the different axes of research that come together to make the contribution in this work. The experiments are thorough and show a clear advantage over the baselines. The baselines are well selected and the differences between them and the new approach is clearly explained. The tables and figures are easy to understand.\n\nAs a suggestion for improvement, the results in the supplement regarding runtime performance are significant and should be included in the main paper with extra space on publication. In addition, the limitation regarding localization and execution should be ultimately included in the main paper. The immediate next question regarding the impact and significance of this work is the ability to run it effectively on a physical robot in real time since no hardware experiments were provided. Evidence in this benchmarking is sufficient to convince me that it would be possible, and further discussion regarding the execution and localization limitations would help inform extensions of this work.\n\nThere is a missing parenthesis in equation 7.\n\nQUESTIONS:\nI am confused about the notation around when and how the top k probability class values are cropped in the semantic vector. It seems to me that \\tilde{P} is the cropped semantic vector and P is the uncropped version. However, \\tilde{P} is only used once in the method section, but it seems that the vector would necessarily be cropped prior to entropy calculations during . Could you clarify?\n\nDid you perform hyperparameter tuning to find the chosen values of each lambda in equation 5? Is the method very sensitive to the choice of hyperparameters?\n\nDo you think that your method could be extended in future work to incorporate open vocabulary semantic segmentation models? Or would it require rethinking the entire approach?\n\nLIMITATIONS:\nYes, particularly in the supplement.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an active semantic mapping technique based on the ActiveGAMER active mapping technique.  The paper proposes a way to extend the Gaussian splatting map representation to include a sparse representation of the semantic information associated with a Gaussian component in the map, a way to update the semantic information using an uncertain measurement, as wel as incorporating the semantic information in the map into an exploration criterion used to determine the next best view used for active mapping.  The proposed approach is tested using a number of simulated scenarios, and evaluated against a number of different competing methods; it achieves good results that compares well with that of competing methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Quality:\nThe proposed approach can be viewed as a fairly simple addition of semantic representation to an existing active mapping approach.  This is done fairly well and is supported by good results; however, this aspect of the paper has its weaknesses:\n- One of the main contributions of the paper is the augmentation of a Gaussian splatting map representation with a distribution over semantic labels.  This distribution is approximated to reduce the storage complexity of the representation.  The approximation is a very simple one: The $k$ most likely labels as seen *from the initial observation* (line 191) are stored and updated.  This will most likely cause the proposed approach to be brittle, since it cannot recover from a case where the correct semantic label is incorrectly discarded based on the initial observation (it conceivable that this could regularly happen in certain types of scenarios).  An obvious alternative representation is to retain a distribution over all the semantic labels, but to group all the labels with a low likelihood into a \"super category\"; this would allow a similar reduction in storage requirement as that of the proposed approach, but would allow the mapping technique to recover from inaccurate semantic observations in the initial view.  \n- Another contribution is the incorporation of the semantic map representation into the exploration criterion used for active mapping.  However, this is done very simply (Eq. 8) and is unmotivated; this aspect of the paper could have benefitted from a more principled development.  \n- The proposed approach is a simple adaption of the active mapping technique ActiveGAMER (ref. 74 in paper; see \"Originality\" for a discussion).  As a result, the proposed approach inherits weaknesses inherent to ActiveGAMER.  For example, the proposed approach seems to perform active mapping by calculating the next-best view, and then navigating to this view (Fig. 1).  This is quite a rudimentary form of active mapping, and it would have been better to plan a trajectory based on the expected information gain *along the trajectory*.  It is also not clear how the proposed approach could be extended to function in real-world scenarios, where it might not be possible to reach the calculated next-best view, and even if it is reachable, the very simple motion cost model used would probably be inappropriate.\n\nThe proposed approach was tested in a number of different ways and evaluated against several other competing methods.  However, this aspect of the paper also has some minor weaknesses:\n- The proposed approach is only tested in simulated scenarios.  Although the simulation environments are widely used and photorealistic, the lack of testing on real-world data causes uncertainty about the applicability of the proposed approach to real-world scenarios.  Testing active mapping in the real world would have been a significant undertaking though.  \n- The paper evaluates the proposed approach against a range of different techniques.  However, some of the competing techniques address a different problem (SLAM vs. mapping, and active *semantic* mapping vs. active mapping), and some of the competing techniques are not evaluated in exactly the same way (see discussions under \"Clarity\" below).  This makes it difficult to reliably draw conclusions from the results of the experiments.  The paper could have benefitted from a more systematic evaluation, isolating the causes for the differences in results seen by the evaluated methods. For example, since the semantic representation chosen for the proposed approach is one of the main contributions of the paper (see \"Originality\" below), an experiment that directly compares the proposed representation with alternatives would have provided a much better way to evaluate this design choice.  \n\n### Clarity:\nThe paper is mostly well written, clear and neat.  The ideas presented in this paper and how they are related are quickly clear on an overview level after a quick read-through.  However, when reading carefully through the paper, a number of issues with the presentation become clear, including the following:\n- The reasons for reviewing SLAM techniques (lines 76-84 and 118-140) and evaluating against SLAM techniques (Sec. 4) are not clear, since the paper proposes a semantic *mapping* technique.  If the goal is to review and evaluate against the *map representation* of existing SLAM techniques, then it has to be done in a much more motivated and nuanced manner than is done in the paper, since the contexts are different.  The statement in line 84: \"We use SplaTAM [51] as the SLAM backbone of our algorithm,\" which is obviously not accurate, compounds this issue.  For readers not already familiar with the field, it would be beneficial to mention the difference between mapping and SLAM.  \n- Although the related work is discussed extensively and clearly, the relationship between the proposed approach and the related work could have been presented better.  For example, two active semantic mapping techniques, SSMI and TARE (refs. 100 and 101 in the paper) are evaluated against, but they are not included in the literature review of Sec. 2, and they are therefore not characterised at all.  \n- The proposed method uses local and global keyframes (see Fig. 1 and lines 226-230), but their meanings and the distinction between them are never given (the discussion in lines 226-230 provides no help).  \n- The system overview in Fig. 1 shows a path planner that forms part of the exploration module; however, this path planner is not described anywhere in the paper.  \n- The Gaussian splatting representation used in the proposed approach, as well as the loss function used for training are discussed in Sec. 3.1, but the training procedure is not discussed at all.  For example, it is unclear how Gaussian components are generated and how their parameters are subsequently determined.   \n- There are several aspects of the map representation that is unclear or ambiguous, including:\n  - According to the rendering procedure in Eq. 1, *every* Gaussian component in the map will infuence *every* pixel in the image.  There is presumably some cut-off where the effect of a Gaussian component below a certain threshold is excluded, but this is not mentioned at all. \n  - The map representation seems to require RGB-D inputs (see lines 176 and 179), but this requirement is never explicitly stated, and it is not clear whether the proposed approach can be applied to RGB images.  \n  - The meaning of $f_i(\\mathbf{p})$ is not given (line 177), and consequently Eq. 1 cannot be interpreted with certainty.  \n  - For the definition of the semantic loss (lines 204-219), the OneFormer observations are used as ground truth and the projected label distributions as predictions.  To remove uncertain \"ground truth\" labels, a mask is defined based on the entropy of the distribution over the labels (line 207), but this entropy is then defined in terms of the *predicted* label distribution (Eq. 4), which does not make sense to me.  \n- In the definition of the geometric exploration criterion (Eq. 6), the pixels that have a nonzero silhouette value are counted.  However, according to the definition of the silhouette in Eq. 1, no pixel will have a silhouette value that is exactly zero (except at initialisation), and the geometric exploration criterion in Eq. 6 therefore simply counts the number of pixels in the image (which does not make sense).  There is presumably some cut-off where the silhouette projection to a pixel that falls below a certain threshold is considered to be zero, but this is not discussed at all.  The motivation for the way in which the geometric and semantic exploration criteria are combined (Eq. 8) is also not given.  \n- The results initially appear to be quite good, but at closer inspection, there are some issues with the experimental setup and results that are not properly discussed, and which create some uncertainty.  These include:\n  - It is unclear why the F-1 scores in Table 1 are only give for some entries.\n  - It appears that results of the first 4 rows of Table 1 have been lifted from the SGS-SLAM paper (ref. 7 in the paper), but this is not mentioned, and there is therefore questions about whether the experimental setups are the same and consequently whether the comparison is fair.  \n  - The first 4 rows of Table 1 are all results of SLAM methods, which addresses a different (although related) problem than that of the proposed approach, which addesses the mapping problem.  An assumption of the mapping problem is that the camera poses are known with certainty, whereas for the SLAM problem, the poses are estimated.  The availability of exact knowledge of the camera poses could possibly be a significantly advantage for the proposed approach when comparing the quality of the maps; however, this fact is not discussed in the interpretation of the results.  \n  - Line 328 seems to say that results for the first 3 rows in the red band in Table 1 have been lifted from the paper by Zhang *et al.* (ref. 92 in the paper); however, it was not clear to me whether this was the case (I could not find the results quoted in Table 1 in the paper by Zhang *et al.*).  In addition, lines 326-328 state that the proposed approach is not evaluated on exactly the same data as the competing methods, and it is consequently uncertain whether any reliable conclusion can be made from a comparison between the results.\n  - Fig. 3 illustrates issues with the predicted semantic labels for SGS-SLAM in one scene and compares it with the ground-truth labels.  However, the predicted labels for the proposed approach for the same scene is not shown, and the relevance of this figure is therefore not clear.  In addition, it is unclear whether the issues are influenced by the pose uncertainty inherent to the SLAM problem, which makes the relevance of the figure even less clear.  \n  - Line 294 states that top-1 and top-3 accuracy results are given, but I could not find it anywhere in the paper.  \n\n### Significance:\nDespite the limited nature of the new ideas presented in this paper (see \"Originality\" below), the good results obtained in the experiments and simplicity of the new ideas give an indication that the ideas will most probably be adopted by the research community.  \n\n### Originality:\nThe technique proposed in this paper is heavily based on the active mapping technique ActiveGAMER (ref. 74 in the paper).  Specifically, the geometric and colour map representation and approach to learning the map, the exploration map and approach to select the next-best view, the path planner, and the keyframe scheme are identical.  Essentially, the proposed approach only adds:\n- a representation for incorporating semantic information into the map,\n- a way to project the semantic information stored in the map to an image and use it to train the map, and \n- the incorporation of the semantic information in the map into an exploration criterion.  \n\nNevertheless, the proposed does indeed seem to be the first active semantic mapping technique based on the Gaussian splatting map representation (lines 349-350).  Good results also support the contribution.  \n\nA note about contextualisation: Although ActiveGAMER is appropriately mentioned and referenced, the extend to which the proposed approach is similar to ActiveGAMER is only clear after a careful read of the paper.  A clearer contrast between the two methods would have been better.  \n\n### Other:\nThere are a number of minor issues in the paper, including the following:\n- Line 82: \"TSDF\" is undefined.  \n- Lines 124, 126 and 141: Quotation marks should be formatted better.\n- Line 141 states that \"All the above approaches are \"passive\"\", but this is not true since active mapping is discussed in lines 85-105.  \n- Line 176: $c_i$ should be $\\mathbf{c}_i$, and $d_i$ is undefined.  \n- Eq. 7: A bracket is missing.  \n- Table 1: The meaning of \"GT\" is undefined; the reader has to guess that this refers to the ground truth.  \n- Lines 320-321: \"and introduce arbitrary\" -- sentence incomplete\n\nQUESTIONS:\n- Are there any aspects of this review that are mischaracterisations of the proposed approach or the presentation thereof?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an active mapping system based on 3D Gaussian splatting called Active Semantic Gaussian Mapping (ActiveSGM).\nActiveSGM represents the scene using 3D Gaussians (as opposed to NERFs) which is a new contribution. ActiveSGM infers the semantics of the visible surfaces and combines a geometric and semantic uncertainty criterion for planning exploratory actions to visit frontiers. ActiveSGM operates under a closed vocabulary assumption and shows robustness to noisy semantic predictions by leveraging predictions from a pre-trained model (relevant in real world systems) and balances memory constraints when dealing with many semantic classes in a scene by using the top-k semantic classes instead of all of them.\nAnother key part of the paper is using 3D Gaussians as a representation of the scene instead of NERFs. This is relatively new and the addition of semantics is a novel contribution to such systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper describes a method for active dense mapping using 3D Gaussians as the representation. In addition, ActiveSGM is demonstrated with model predictions (without assuming ground truth semantic labels) and hence describes a good proof-of-concept for utilizing 3D Gaussians and semantic information for active mapping\nThe details of ActiveSGM are easy to follow\nWeaknesses\nMy main criticism is related to the writing in the experiments section of the paper - as a reader I found it hard to grasp what components of ActiveSGM led to better performance over baselines in the individual experiments. I will elaborate on these in the Questions section.\n\nQuality\nThe paper explains the components of the ActiveSGM pipeline well and is easy to follow.\nThe explanation of the exploration heuristics is also fairly intuitive and well explained.\nThe evaluation metrics and qualitative examples highlight capabilities of the proposed method\nClarity\nThe description of the ActiveSGM algorithm is clear. However, when I read the paper, it seems like I could not fully grasp the similarities and differences to previous methods which led to the improvements - a way to mitigate it might be to explicitly talk about the similarities and differences in the components (semantic representation, 3DGS vs NERF, ground truth vs semantic predictions, etc) while explaining the results from the experiments. In some cases the information is there but currently does seem to be emphasized enough. For example, the section on 3D Reconstruction and Novel View Synthesis shows how AvticeSGM is better than ActiveGAMER. But through the paper, I only understood the similarities to ActiveGAMER in terms of objectives and the fact that ActiveSGM does not have photometric refinement stage. So what are some reasons ActiveSGM performed better than ActiveGAMER?\nSignificance\nThe paper presents a method for leveraging 3D Gaussians as the scene representation for active mapping. ActiveSGM also incorporates a sparse representation of semantics and demonstrates active mapping with semantic predictions instead of utilizing ground truth semantic labels.\nOriginality\nThe method is original.\nThis paper on using NERFs for active perception seems relevant to the problem at hand (Active Perception using Neural Radiance Fields https://arxiv.org/abs/2310.09892). They try to maximize a predictive information objective. While they assume ground truth semantic labels, this paper could be an important related work. It would be nice to understand how this fits into the current work and an explanation of the benefits / drawbacks of this compared to ActiveSGM would be helpful to provide more clarity in the paper\n\nQUESTIONS:\n4.2 ReplicaSLAM\nWhat part of the SGS protocol are we talking about here? Is there an active mapping component in this experiment?\nIf we’re just considering a pre-defined trajectory, then only point 1 (using OneFormer instead of ground truth) seems to be different from SGS-SLAM. The other points are only related to training evaluation and training data. In that case, what component of ActiveSGM are the authors trying to highlight with this experiment?\nIf we are considering an active mapping, then how are the number of steps coming into play - because SLAM systems would work with a pre-defined trajectory. So I am unable to understand the purpose of “steps”\nWhat does the training refer to here? Does it refer to fine-tuning the OneFormer model? Based on my understanding, all the components mentioned in section 3 talk about the deployment phase of ActiveSGM, so what does training on 1/3 rd of the data mean for ActiveSGM. Additionally, while giving ActiveSGM a more difficult evaluation in terms of less training data and evaluation on novel views seems interesting, not providing the results while considering the same amount of data raises questions on the ceiling of the ActiveSGM performance - how much better is it compared to the baselines if given all the training data and evaluated on training scenarios or does the performance saturate. And what are the benefits compared to the baselines when the baselines are evaluated on the novel views and ActiveSGM is evaluated in novel views. The current set of of experiments shows different conditions for baselines and ActiveSGM and warrants more controlled evaluation conditions - the current experiment reads like the following - “We gave ActiveSGM less training data and also evaluated on novel views and it performs similar to some baselines and worse than the best” - it is important to have the apples-to-apples in the testing conditions to properly appreciate the system. Alternatively, if there are some holes in my understanding of the experimental setup then please point it out.\n4.2 Replica (Novel Views)\nActiveSGM without exploration performs better than SGS-SLAM, thereby showing the benefits of the sparse representation.\nIt is hard to appreciate the benefits of the sparse representation because the paragraph does not properly emphasize the differences between ActiveSGM and SGS-SLAM. Is the following elaboration appropriate - “ActiveSGM without exploration is the same as using a predefined trajectory and doing SLAM. This is very similar to SGS-SLAM but SGS-SLAM compresses semantic information to r,g,b while ActiveSGM uses the sparse representation. ActiveSGM outperforms SGS-SLAM and hence shows the benefits of the sparse representation over compressing semantics into r, g, b?”. If so, then making this more clear will improve the clarity of the paper. If not, please explain how the authors arrive at that conclusion.\n4.2 MP3D\nThe authors mention that compared to the baselines, ActiveSGM does not rely on ground truth semantic labels and still performs better. Based on the content in the paragraph, it is hard to understand what choices of ActiveSGM compared to the baselines lead to the improved performance. Is it the 3D Gaussian representation? Is it the exploration criterion?\nWhile the individual components of ActiveSGM are explained in Section 3, not emphasizing the differences between ActiveSGM and the baselines in the experiments section makes it hard to understand “why” or “what components” might be contributing to better performance (especially for reader who aren’t extremely well versed with all the baselines)\nIt can be hard to provide an exact answer but a proper discussion on what changes and how that affected the results would provide clarity to readers.\nMP3D experiment (NOT CRITICAL FOR REBUTTAL) - Did the authors consider using ground truth labels for ActiveSGM - would the performance get better or\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-14"}