CSO: Constraint-guided Space Optimization for Active Scene Mapping

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 15:37:50

Abstract: Simultaneously mapping and exploring a complex unknown scene is an NP-hard problem, which is still challenging with the rapid development of deep learning techniques. We present CSO, a deep reinforcement learning-based framework for efficient active scene mapping. Constraint-guided space optimization is adopted for both state and critic space to reduce the difficulty of finding the global optimal explore path and avoid long-distance round trips while exploring. We first take the frontiers-based entropy as the input constraint with the raw observation into the network, which guides the training start from imitating the local greedy searching. However, the entropy-based optimization can easily get stuck with few local optimal or cause inefficient round trips since the entropy space and the real world do not share the same metric. Inspired by constrained reinforcement learning, we then introduce an action mask-based optimization constraint to align the metric of these two spaces. Exploration optimization in aligned spaces can avoid long-distance round trips more effectively. We evaluate our method with a ground robot in 29 complex indoor scenes with different scales. Our method can perform 19.16\% more exploration efficiency and 3.12\% more exploration completeness on average compared to the state-of-the-art alternatives. We also implement our method in real-world scenes that can efficiently explore an area of 649 $m^2$. The experiment video can be found in the supplementary material.

Main paper: 
 introduction: 
 Building a complete map from an unknown indoor environment based on robot scanning is critical for many applications in computer vision and robotic communities. However, it is time-consuming and inconvenient if the scanning is human-operated. Introducing artificial intelligence to guide the robot scanning automatically is then proposed by many previous works [9, 16, 17, 27, 42, 43, 45] , which form the optimization problem of active scene mapping.The goal of active mapping is finding the shortest exploration path to perform the complete scanning. However, solutions based on global planning theory, like the Travelling Salesman Problem (TSP) [30] , formulate it as an NP-hard problem [26] , which makes 1), (2) , and (3) , where points 𝐴, 𝐵, 𝐶, and 𝐷 represent the robot's position, the optimal position, a position with higher frontier entropy, and a position farther from the robot, respectively. When two metric spaces are aligned, the distances between any two points in these spaces should be similar or exhibit proportional scaling. As shown in (4), we select 𝐵 𝑖 , 𝐶 𝑖 , 𝐷 𝑖 from spaces (1), ( 2), (3) to construct 3 triangles △𝐵 𝑖 𝐶 𝑖 𝐷 𝑖 , 𝑖 = 1, 2, 3 individually. It is evident that𝐵 2 𝐶 2 𝐵 3 𝐶 3 ≈ 𝐶 2 𝐷 2 𝐶 3 𝐷 3 ≈ 𝐵 2 𝐷 2 𝐵 3 𝐷 3 , yet 𝐵 1 𝐶 1 𝐵 2 𝐶 2 𝐶 1 𝐷 1 𝐶 2 𝐷 2 𝐵 1 𝐷 1𝐵 2 𝐷 2 , which suggests that space (3) aligns with (2) but not with (1) . The final result is depicted in (c), the robot prefers the closer point 𝐵 with lower entropy instead of the farther point 𝐶 with higher entropy, thus avoiding longdistance round trips(orange dashed path, 𝐴 → 𝐶 → 𝐵) and follow a more rational path (green solid path, 𝐴 → 𝐵 → 𝐶).it unsolvable if the environment is large and complex. Hybridstage optimization [6, 34] , which combines the greedy search and global planning, is then proposed for map reconstruction of large scenes. The core idea is that global planning is only adopted at the coarse-grained level of the mapping, while explicit local exploring is performed by greedy search based on information entropy. This solution shows good time efficiency even if the unknown environment is complex. However, the two optimization stages are relatively independent. Local exploration sometimes would drive global planning into a local optimal.The recently developed reinforcement learning techniques provide another efficient solution for this NP-hard problem. A critic network is introduced to replace the handcrafted rules to estimate the importance of each exploration destination. A correlated action network is adopted to perform the final planning. [43] takes advantage of the recent reinforcement learning solutions and neural graph networks for more efficient and complete map construction with multi-robots, whose excellence is that the whole optimization is end-to-end. However, raw scanning data is not informative enough if we only have a single robot for mapping. Bad mapping initials and noised input without cross-validation between robots highly limit the performance of previous multi-robot-based methods, which sometimes leads to long-distance round trips during global planning in a complex scene.Additionally, the misalignment of space metric between the environment and corresponding critic value space leads to random stuck under exceptional circumstances since the robot chooses the next exploration goal based on the critic value estimated for each position. More specifically, the misalignment would make goal positions with similar critic values distribute around the whole scene. This situation would keep the robot on the run with round trips between these locations. Therefore, aligning the metric between these two spaces, which gives the critic network more specific guidance, is critical for efficient active mapping with a single robot.This paper proposes a novel reinforcement learning-based active mapping approach with constraint-guided space optimization for both state and critic space. Specifically, we first introduce a frontiersbased entropy as the input constraint with the raw observation into the network to boost the local optimization. The advantage of this is that the heuristic-based entropy can cover the bad planning given by the critic network with incomplete initials, which can significantly enhance the mapping efficiency at the beginning. It is important to emphasize that this extra input would not mess up the global planning with enough observation like the previous hybrid-stage optimization approaches. Since the entropy is given implicitly through the network, the two-stage optimizations are highly correlated.The second stage of space optimization is performed by the action network, which takes the outputs of the critic network to give the global planning. We introduce an action mask strategy to solve the misalignment problem of the space metric we mentioned above. It is not easy to align the metric without explicit supervision. Inspired by constrained reinforcement learning [21, 23, 44] , we utilize the relevance of the critic network and action network to perform the alignment. At the beginning of training, we constrain the action network through a local action mask with a limited size near the robot. In response, the critic network tends to tune down the estimated scores for these long-distance goals since the action network is not selecting them. And then the critic value space will gradually align with the real-world space, as shown in figure 1 .We evaluate our method with a ground robot in 29 complex indoor scenes. In the experiments, our method can perform 19.16% more exploration efficiency and 3.12% more exploration completeness on average compared to the state-of-the-art alternatives. Furthermore, the experiments demonstrate that our method also has significantly better mapping performance on more complex and larger-scale scenes thanks to the frontiers-based entropy design. Some visual examples are given in figure 6 , which demonstrate the proposed action mask's effectiveness in avoiding long-distance round trips during active mapping. We also implement our method with a ground robot in 3 real-world scenes that can efficiently explore areas from 170 𝑚 2 to 649 𝑚 2 , as shown in figure 7 . In summary, the contributions of this paper include the following:• We introduce a frontiers-based entropy to constrain the network input, which can significantly improve training efficiency and mapping performance. This design explored a novel way to make integration of the heuristic-based method and advanced reinforcement learning-based techniques. • We propose an action mask constraint to guide the network to do efficient global planning by aligning the metric between the critic value space and the real-world space. This design is demonstrated to be effective for avoiding round trips during the exploration, which can significantly improve the efficiency of active mapping. • The proposed constraint-guided space optimization-based active scene mapping outperforms state-of-the-art alternatives with indoor scenes, which gains 19.16% more exploration efficiency and 3.12% more exploration completeness on the Matterport3D dataset. 
related work: 
 Traditional Heuristic Method: The pioneering work [42] first proposed the concept of the frontier for active mapping, i.e., the boundary areas between explored free space and unexplored space, aiming to guide the robot to the frontiers until the entire space is observed. Many subsequent works [2, 11, 13, 17, 20, 37, 46, 47] achieve both 2D and 3D active mapping based on this concept. In addition, [6, 16, 19] have also achieved good performance based on Rapidly Exploring Random-Trees [24] , Travelling Salesman Problem [30] , etc. These methods work well for active mapping. However, most of them either greedily select frontiers, making it difficult to obtain an approximate global optimal solution, or the planning process is time-consuming and computationally expensive.Learning-Based Method. [9, 10, 43] employ convolution neural network or graph neural network [22] to regress the goal position by maximizing the long-term value via reinforcement learning, providing less planning time and approximate optimal strategy. These works directly use raw data as state input without further data processing, which will improve exploration efficiency. In addition, [7] successfully uses an attention-based deep reinforcement learning approach for active mapping. However, they only conduct experiments in perfect and synthetic 2D scene maps, so for realistic scene data such as Matterport3D [8] , Gibson [41] , and real-world, it may cause the performance drop due to the lack of sim-to-real.Information-Theoretic Methods Another popular way for active mapping is using the information theory. By calculating the information gain or uncertainty value [3, 14, 15, 19, 29, 31, 32] , the robot chooses actions that can maximize information gain or reduce uncertainty of the scene. The earliest information-theoretic strategies are those proposed by [40] and [18] . [1, 4, 38, 39] maximize information gain over the next few actions. And these works can also be extended to solve multi-robot tasks [5, 12, 28] . This planning cycle is iteratively implemented until the termination criteria are triggered.State 
method 3.1 problem statement and formulation: 
 The goal of active mapping is to let the robot automatically decide its next action based on its local observations in an unknown environment and gradually generate a globally optimal shortest exploration path, which can completely scan the scene and build a complete map. This optimization process can be defined as:EQUATION𝑠.𝑡 . 𝐸 - ∑︁ 𝑡 𝐶 (𝜔 𝑡 ) = 0where 𝐿 is the exploration trajectory length, dist(•) is the travel length from location 𝜔 𝑡 to 𝜔 𝑡 -1 , 𝐶 (•) is the coverage area from a certain scanning location, 𝐸 is the complete environment mapping, 𝜔 𝑡 is the robot's location at 𝑡. Note that 𝐸 and 𝐶 (•) are unknown before the mapping. Thus this optimization can not be solved directly.Since we can not solve the optimization at one step, we formulate equation 1 as a Markov Decision Process (MDP) to solve this illposed problem step-by-step, whose training details can be found in section 3.4. Specifically, the learning process is determined by the tuple: (𝑆, 𝐴, Π, 𝑅), where 𝑆 is the state space constructed from the scanning during the exploration while 𝐴 is the action space of the robot. Note that we use the frontiers of the occupancy map constructed while exploring the scene as the state and action space, which can greatly reduce the space size and make the learning process easier to converge. Furthermore, we propose a frontiersbased entropy as the input constraint for the state space 𝑆, which guides the training start from imitating the local greedy searching. Details about 𝑆 can be found in section 3.2. We then introduce an action mask design as a constraint to align the metric of 𝑆 and the critic space given by Π. Π(𝑎|𝑠) denoting the probability of selecting action a under state 𝑠. Details of action mask design can be found in section 3.3. The overview of our pipeline is shown in figure 2.The state for our framework mainly consists of three parts, the occupancy map 𝑀 𝑡 , the distance map 𝐷 (𝑀 𝑡 , 𝜔 𝑡 ), and the proposed frontiers-based entropy 𝐼 (𝑀 𝑡 ).EQUATIONOccupancy map. Given the observation 𝐶 (𝜔 𝑡 ) from the robot at location 𝜔 𝑡 , a 2D global map from the top-down view of the 3D scene is constructed as the occupancy map. The occupancy map is denoted as 𝑀 𝑡 ∈ [0, 1] 𝑋 ×𝑌 ×2 at time step 𝑡, where 𝑋, 𝑌 are the map size. The two channels indicate the explored and occupied regions separately. Each cell in 𝑀 𝑡 can be classified as one of the three classes, open (explored but not occupied), occupied, and unknown (unexplored). Frontier cells 𝐹 𝑡 ⊂ 𝑀 𝑡 are defined as the open cells whose adjacent to at least one unknown cell.Distance map. Given the current position 𝜔 𝑡 and the constructed occupancy map 𝑀 𝑡 build by the robot while exploring, we further construct a distance map 𝐷 (𝑀 𝑡 , 𝜔 𝑡 ) ∈ R 𝑋 ×𝑌 , in which 𝐷 𝑥,𝑦 (𝑀 𝑡 , 𝜔 𝑡 ) represents the geodesic distance from the location (𝑥, 𝑦) to the robot's position 𝜔 𝑡 .EQUATIONThe geodesic distance dist 𝑀 𝑡 is the shortest distance for traversal between two points in 𝑀 𝑡 without collision. Compared with the Euclidean distance, it can implicitly encode the underlying scene layout information, which gives better guidance for robots to explore. We compute dist 𝑀 𝑡 with the Fast Marching Method [35] .Frontiers-based Entropy. Besides the raw data 𝑀 𝑡 and 𝐷 (𝑀 𝑡 , 𝜔 𝑡 ), we would like to introduce some high-level information as a constraint to reduce the searching space while the 𝑀 𝑡 is highly incomplete. Heuristically, the frontier cell 𝑓 ∈ 𝑀 𝑡 indicates the potential next best goal to explore since its surrounding is information incomplete. And the frontiers in a scene usually do not exist alone, but have the characteristics of small-scale aggregation and large-scale dispersion. Inspired by this, we propose a frontiers-based entropy𝐼 to encode such information as input to constrain the network searching, which can be described as:EQUATIONwhere 𝐼 𝑥,𝑦 (𝑀 𝑡 ) denotes the number of frontiers calculated by the Information Aggregation module based on the 𝛾 ×𝛾 submap centered on (𝑥, 𝑦) in 𝑀 𝑡 , as shown in figure 3 . By doing this, the information contained in each location, in addition to its map coordinates (𝑥, 𝑦), also contains statistical information within its neighborhood. Note that 𝐼 𝑥,𝑦 (𝑀 𝑡 ) will give (𝑥, 𝑦) a larger information entropy if it is surrounded by more frontiers. For the discrete frontiers or noise points, a smaller information entropy can appropriately reduce its impact on global optimization.In the reinforcement learning framework, the robot chooses the next exploration goal 𝜔 𝑡 +1 based on 𝜔 𝑡 +1 = arg max 𝑓 ∈𝐹 𝑡 Π(𝑎|𝑠 (𝜔 𝑡 ). The action space 𝐴 is reduced to selecting a frontier 𝑓 from 𝐹 𝑡 . If the world space 𝑀 𝑡 is not aligned with the space metric of Π(𝑎|𝑠 (𝜔 𝑡 )), it will cause the robot to randomly get stuck or turn back and forth under exceptional circumstances, as shown in figure 1 .Therefore, aligning the metric between 𝑀 𝑡 and Π(𝑎|𝑠 (𝜔 𝑡 ), which gives the critic network more specific guidance, is critical for efficient active mapping with a single robot. Since there is no additional supervision information, we introduce an action mask strategy to solve the misalignment problem of the space metric. Specifically, we design two action masks: the valid distance mask and the stuck mask, to filter the actions in the action space of the global policy, and constrain action sampling to be performed within the valid action space, rather than the entire action space. By doing this, the influence of abnormal data in the network learning process can be reduced, and the network learning rate can be accelerated at the same time so that 𝑀 𝑡 and Π(𝑎|𝑠 (𝜔 𝑡 ) can be aligned.Valid Distance Mask. We first design a valid distance mask to filter some obviously invalid goals in the action space. When the robot explores a large-scale complex scene, a reasonable and efficient exploration method should make the robot gradually explore the scene instead of wandering back and forth between multiple goals. We want the better actions given by Π(𝑎|𝑠 (𝜔 𝑡 )) distributed in a reasonable range around 𝜔 𝑡 to ensure the gradual exploration. This heuristic property also implicitly requires the metric of 𝑀 𝑡 and Π(𝑎|𝑠 (𝜔 𝑡 )) to be aligned.To ensure the constraint is appropriate, not too strict or loose, we filter the action space based on the geodesic distance information from 𝜔 𝑡 to each potential goal. Specifically, we set 2 thresholds, 𝛽 near and 𝛽 far . For the potential goals that exceed the threshold range [𝛽 near , 𝛽 far ] of 𝜔 𝑡 , we set their sampling probability to 0 to ensure that they will not be sampled, and only sample frontiers within the valid distance. The masked Π mask (𝑓 |𝑠 (𝜔 𝑡 )) is given as:EQUATIONwhere Π mask (𝑓 |𝑠 (𝜔 𝑡 )) represents the masked probability of the action to select 𝑓 as the next exploration goal, 𝑑𝑖𝑠𝑡 𝑀 𝑡 (𝑓 , 𝜔 𝑡 ) is the geodesic distance from robot position 𝜔 𝑡 to the 𝑓 , and Π(𝑓 |𝑠 (𝜔 𝑡 )) is the original probability distribution from the network. We set Calculate the max moving length in the last 3 steps:𝛽 near = 0.𝑙 max = max 𝑖 ∈{𝑡 -1,𝑡 -2} 𝑑𝑖𝑠𝑡 𝑀𝑡 (𝜔 𝑖 , 𝜔 𝑖+1 )Calculate the max area increment in the last 3 steps:𝑐 max = max 𝑖 ∈{𝑡 -1,𝑡 -2} ∥𝑀 𝑖 -𝑀 𝑖+1 ∥ Given 3 thresholds 𝛼 1 = 1, 𝛼 2 = 1 and 𝛼 3 = 1 if 𝑙 max < 𝛼 1 and 𝑐 max < 𝛼 2 thenfor Location 𝜔 𝑎 given by 𝑎:min 𝑖 ∈{𝑡,𝑡 -1,𝑡 -2} 𝑑𝑖𝑠𝑡 𝑀𝑡 (𝜔 𝑖 , 𝜔 𝑎 ) < 𝛼 3 do Π mask (𝑎 |𝑠 (𝜔 𝑡 ) = 0 end endStuck Mask. Besides the valid distance mask, we further design a stuck mask, which is used to filter the actions from the action space that will cause the robot to continue being stuck. To be specific, we first determine whether the robot is stuck by collecting the robot's long-term goals, exploration areas, and moving distances in the past 3 time steps. We consider the robot to be stuck if the maximum moving length is less than the threshold 𝛼 1 = 1 and the maximum area increment is less than the threshold 𝛼 2 = 1. Then, we calculate the distance from the candidate actions to the past 3 long-term goals. Only when the minimum distance is greater than the threshold 𝛼 3 = 1, we consider the action to be a reasonable action that has a probability of being selected as the long-term goal. By doing this, a new long-term goal that is different from the historical goals will be chosen, which can help the robot get out of stuck. Details of the stuck mask operation can be found in Algorithm 1.We use the off-policy learning approach Proximal Policy Optimization (PPO) [33] as the policy optimizer. The actor is trained to learn a policy network that outputs the sampling probability of each action. Graph neural network [22] is adopted as the encoder to perform feature extraction and fusion. The critic trains a state-value network that predicts the state value 𝑉 (𝑠 (𝜔 𝑡 )) to indicate how much reward is earned from the current state, which is adopted to train our actor network. Details of the network can be found in the supplementary materials.Graph Neural Network-based Encoder. We first construct a graph 𝐺 (𝐹 𝑡 , Ω 𝑡 ) based on the frontiers 𝐹 𝑡 and the explored path Ω 𝑡 = {𝜔 0 , ..., 𝜔 𝑡 } to represent the context of the current scene. It establishes the correspondences between the robot and frontier nodes extracted from the constructed occupancy map 𝑀 𝑡 . We distribute the information given by state 𝑠 (𝜔 𝑡 ) into nodes and edges of 𝐺 (𝐹 𝑡 , 𝜔 𝑡 ). For each node 𝑛 𝑖 , the input feature 𝑓 (𝑛 𝑖 ) ∈ R 5 includes the (𝑥, 𝑦) coordinates in 𝑀 𝑡 , the semantic label that indicates 𝑛 𝑖 ∈ 𝐹 𝑡 or 𝑛 𝑖 ∈ Ω 𝑡 , the history label that indicates 𝑛 𝑖 is 𝜔 𝑡 or explored nodes 𝑛 𝑖 ∈ {𝜔 0 , ..., 𝜔 𝑡 -1 } and the proposed frontiers-based entropy 𝐼 𝑛 𝑖 (𝑀 𝑡 ). The edge feature 𝑓 (𝑛 𝑖 , 𝑛 𝑗 ) ∈ R 32 is given by a multi-layer perception(MLP) with 𝑙 𝑖 𝑗 ∈ R 1 indicates the geodesic distance from node 𝑛 𝑗 to node 𝑛 𝑖 . Then we feed these node and edge features into the GNN network, which is our actor network, for feature transfer and output a set of scores. Based on these scores, we compute the probability Π mask (𝑓 |𝑠 (𝜔 𝑡 )) of each action, followed by action mask-based action sampling.Our training goal is to maximize the accumulated reward function 𝐺 (𝜔 𝑡 |Ω 𝑡 -1 ). For the active mapping task, the goal is to pursue high-time efficiency and map completeness. To achieve this goal, we use an efficiency reward 𝑅 step and a coverage reward 𝑅 𝑡 coverage . The efficiency reward 𝑅 step punishes unnecessary time steps to encourage the robot for more efficient exploration. And the coverage reward 𝑅 𝑡 coverage = ∥𝑀 𝑡 ∥ -∥𝑀 𝑡 -1 ∥ is defined as the coverage increment at time step 𝑡, where the coverage ∥𝑀 𝑡 ∥ is the area of the open space in the occupancy map 𝑀 𝑡 . Then the complete reward function formula is defined asEQUATIONwhere 𝜆 𝑐 is the hyper-parameter to balance these two rewards. 
experiment 4.1 experimental setup: 
 We test our method in both simulated and real-world environments.For experiments in simulation, our experiments are conducted on the Matterport3D [8] dataset using the iGibson [25, 36] simulator.For experiments in the real world, we deploy a LIMO robot to explore 3 distinct real-world scenes, as shown in figure 7 . Detailed implementations and the video of real-world experiments can be found in the supplementary material.Data processing. We conducted simulation experiments on the Matterport3D dataset [8] containing 90 realistic indoor scenes using the iGibson simulator [25, 36] . We followed the standard train/val/test split of Matterport3D and further split them into four scales: small(< 30𝑚 2 ), middle(30 -100𝑚 2 ), large(100 -260𝑚 2 ) and super large(> 260𝑚 2 ) according to their traversable areas. Details of the splits can be found in the supplementary materials.Termination criteria. We consider exploration complete when the coverage is greater than 99% while the unexplored area is less than 1𝑚 2 , or there are no accessible frontiers in the environment. Meanwhile, we also set the maximum exploration steps as 𝑛 = 3000. If the exploration step exceeds 𝑛, we terminate exploration anyway.Evaluation metrics. We evaluate the map completeness via the coverage(%) that calculates the percentage of explored open space over the ground truth open space in the environment, and the exploration area(𝑚 2 ) that calculates the area of explored open space in the environment. In addition, we use the ratio of exploration area to the time(𝑠𝑡𝑒𝑝) as the metric to evaluate the scene exploration efficiency(𝑚 2 /𝑠𝑡𝑒𝑝), which represents the average area scanned at each step while exploring a scene.We compare our method with learning-based methods (ANS, UPEN, ARIADNE, NeuralCoMapping) and traditional heuristics methods (Greedy-Info, Greedy-Dist), details of each method are as follows:• Active Neural SLAM(ANS) [9] . ANS learns a policy that uses an egocentric local map and a geocentric global map as input and regresses the goal estimation for path planning. • UPEN [19] . UPEN learns the occupancy priors over indoor maps to generate occupancy maps beyond the field-of-view of the agent and then leverages the model uncertainty over the generated areas to formulate path selection policies for the task of interest. In this work, the Rapidly Exploring Random-Trees(RRT) [24] is used for path planning. • ARIADNE [7] . ARiADNE proposes a reinforcement learning approach that relies on attention-based deep neural networks for autonomous exploration. However, they only conduct experiments in perfect and synthetic 2D scene maps instead of realistic 3D simulated scenes such as Matterport3D [8] . Therefore, we pre-generate 2D maps of all scenes with the same initialization information and then conduct tests. • NeuralCoMapping [43] . Following the recent graph neuralnetwork-based reinforcement learning solution NeuralCoMapping, we adapt their work from a multi-robot active mapping to a single-robot for comparison with our approach. • Greedy-Info and Greedy-Dist [42] . Based on the frontiers proposed by [42] , we further design two heuristics methods, Greedy-Info and Greedy-Dist, as two variants of our method and use them as two alternatives for comparison based on the greedy strategies. These two methods choose the longterm goal in a greedy manner based on the geodesic distance or frontiers-based entropy. Detailed implementations can be found in the supplementary material. Quantitative Comparison with State-of-the-art Alternatives. We compared our algorithm with all alternatives in 29 prior unknown scenes consisting of the Matterport3D test and val dataset according to the standard split. We conducted 5 tests for each scene, with all methods starting with the same position and orientation, and the final statistics are shown in table 1. It's obvious that our method outperforms all alternatives in terms of exploration completeness, time steps, and efficiency, demonstrating the superiority of our proposed method. Specifically, our method improves the exploration efficiency by about 19.16% and coverage by about 3.12% compared with the state-of-the-art method NeuralCoMapping [43] .It can also be found from figure 5 that the superiority of our method in exploration efficiency over alternatives becomes more significant as the scene scale increases. This is because our method effectively reduces long-distance round trips while exploring and enables more efficient exploration. It is not a visible effect in small scenes, but as the scene scale increases, the effect of long-distance round trips on exploration efficiency becomes increasingly obvious. Therefore, the larger the scale of the scene, the more pronounced the superiority of our method becomes.We also compared the planning time of all methods while exploring. Compared with the time-consuming traditional method like RRT(0.878𝑠), our planning time(0.024𝑠) is only 2.73% of it. Due to using frontiers-based entropy and action mask operations, our method causes a longer planning time than other learningbased methods like NerualCoMapping(0.019𝑠) and simple greedy strategies(0.015𝑠). Nevertheless, our method still maintains the same level while achieving better performance. Detailed planning times for all methods can be found in the supplementary material.Visual Comparison with State-of-the-art Alternatives. Figure 6 shows the visualization results of all methods in scenes with different scales from the Matterport3D dataset, where the scanning completeness and exploration paths among different alternatives can be intuitively compared. We also record the exploration coverage curves for each scene corresponding to figure 6 , as shown in figure 4 . It can be seen from figure 6 that our method outperforms others in both exploration coverage and path rationality. Meanwhile, our exploration curves in figure 4 are closer to the upper left, meaning our method can explore scenes efficiently. Specifically, for some small and simple scenes (a)-(c) where the whole scene can be explored within 𝑛 = 3000 steps, our exploration paths are more reasonable and there are no long-distance round trips or serious stuck situations while exploring. As for large and complex scenes (d)-(h) that all methods cannot explore the whole scene within 𝑛 = 3000 steps, our method can explore more regions with a more reasonable path and achieve higher exploration completeness. This is because the critic value space is aligned with the real-world geodesic distance space, preventing the robot from falling into local optimums and long-distance round trips.In comparison, ANS [9] uses the entire map as the policy input and outputs any position in the map as the next long-term goal without any constraints, which makes it easy to select goals that will cause long-distance round trips. NeuralCoMapping [43] and ARIADNE [7] enhance exploration performance by constraining the action space to frontiers or neighboring nodes around the robot instead of the entire map. However, the same problem is still inevitable because the distance from frontiers or neighboring nodes to the robot varies across different scenes due to different scene scales and dynamics while exploring. In addition, UPEN [19] uses RRT [24] for path planning based on the occupancy map uncertainty predicted by their model and selects the path that can maximize map uncertainty over candidate paths for exploration. Therefore, it depends heavily on the model predictions. At the same time, path planning based on RRT is also time-consuming. More comparison results and discussions can be found in the supplementary material.We conduct ablation studies to evaluate the effects of introducing two kinds of action masks and frontiers-based entropy on exploration efficiency. The results are shown in table 2. We can find that using either action masks (+AM) or frontiers-based entropy (+Info) can improve exploration efficiency. This is because either of them can provide positive guidance for network learning, guiding the strategy to choose the actions with larger entropy value or filter out some obvious invalid actions, thus improving the exploration efficiency. However, these methods are still myopic. Especially when exploring large-scale scenes that need to maximize longterm rewards, it is still difficult to further improve the exploration efficiency by simply using either of these two approaches.Table 2 : Ablation studies of our method, where "baseline" is the NeuralCoMapping [43] , "+SM" indicates using stuck mask, "+VDM" indicates using valid distance mask, "+AM" means that both stuck mask and valid distance mask are used, and "+Info" indicates using frontiers-based entropy. Another important thing to note is that solely employing either a stuck mask (+SM) or a valid distance mask (+VDM) will diminish exploration efficiency. Only through the combined utilization of both can exploration efficiency be enhanced. We analyze that if only one type of action masking is performed, the probability of sampling the other type of invalid actions will increase, thus leading to a decrease in exploration efficiency. Only when both action masks are used simultaneously can the policy learn effective action sequences from valid actions, thus maximizing long-term rewards and improving exploration efficiency. 
conclusion: 
 We propose a constraint-guided space optimization for active scene mapping. The space optimization consists of two aspects. We first introduce the transitional heuristic rules to formulate a frontiersbased entropy to constrain the search space, which can significantly improve the exploration efficiency at the beginning of the scanning. The second constraint is introduced by our action space design, which can align the metric of the critic value space and the realworld space. This design can significantly help the robot avoid longdistance round trips while exploring a large and complex scene. The evaluation results also demonstrate the superiority of our method while comparing it with the state-of-the-art alternatives. 
 
 Appendix:  


Decision: This paper proposes a deep RL method for efficient active scene mapping. To tackle the problems of the few local optimal and inefficient round trips in the entropy-based optimization, the paper designs an action mask-based optimization constraint to align the metric of the entropy space and the real world. Extensive experiments in indoor scenes demonstrates its effectiveness over the previous SoTA and a real-world experiment is conducted to show its efficiency.

All three reviewers agree on the acceptance of this paper for its effectiveness shown in the experiment, and the appreciable real-world deployment. Concerns such as the insufficient related work, the lack of novelty, and some format issues are resolved in the rebuttal. After considering all these effort, AC agrees with reviewers that the paper should be accepted. Please revise the paper accordingly based on comments.

Reviews: review: paper_summary: paper_summary: The paper presents a method for efficient exploration of complex unseen environments that uses an action mask-based constraint to align the entropy/metric space with the critic-value space of a reinforcement learning algorithm.
The method is composed of two stages of optimization, the first uses a frontiers-based entropy constraint, while the second uses an action mask strategy to mitigate the space misalignment between real-world metrics and critic value.
strength_weakness: strength_weakness: The paper is engaging and well-written, introducing an interesting method for efficiently exploring complex indoor environments using a robotic agent. The method is comprehensively detailed, ensuring that the code can be easily reproduced. The figures are seamlessly integrated with the text, enhancing the understanding of the method and offering valuable insights during the qualitative evaluation.

The experimental evaluation is comprehensive, thoroughly examining all proposed modules. This extensive evaluation demonstrates the improvement provided by the proposed approach. The qualitative experiments in Figure 6 are very interesting and useful to give an idea of the final behavior of the proposed method compared to the baselines.

The real-world deployment is an appreciable experiment to validate the performance of the method in real-world conditions.
limitations: limitations: The related work section is very brief and could be extended to better frame the proposed method in the current literature on embodied exploration methods. For example, some related work adopting learning-based exploration methods is not included in the paper [A, B]. Moreover, current literature on learning-based methods for exploration is also ignored in the experimental evaluation [31, A].

The exploration results on MP3D contained in Table 1, seem relatively low compared to other work on exploration [31] considering that the time budget is quite high (3000 steps). Can the authors elaborate on this difference?

The overall novelty of the proposed method is limited as the main contribution is given by a mask on the action space of the navigation policy.

A big concern I have about the paper is its suitability to the multimodal community as it seems there is not a clear multimodal aspect in the proposed approach, but I could be convinced otherwise. Could the authors explain how this work could be beneficial to the MM community?

Typos: line 687 -> NerualCoMapping

[A]: Bigazzi, R., Landi, F., Cascianelli, S., Baraldi, L., Cornia, M., & Cucchiara, R. (2022). Focus on impact: indoor exploration with intrinsic motivation. IEEE Robotics and Automation Letters, 7(2), 2985-2992.
[B]: Hu, Y., Geng, J., Wang, C., Keller, J., & Scherer, S. (2023). Off-policy evaluation with online adaptation for robot exploration in challenging environments. IEEE Robotics and Automation Letters.
paper_summary: paper_summary: The paper introduces a deep reinforcement learning-based framework designed to enhance the efficiency and completeness of active scene mapping with a single robot. The framework, named CSO, employs constraint-guided space optimization by integrating a frontiers-based entropy metric and an action mask strategy. The frontiers-based entropy metric prioritizes areas with high information gain to improve local search efficiency, while the action mask strategy aligns the critic value space with the real-world space to avoid redundant long-distance movements. Evaluated in 29 complex indoor scenes and through real-world tests with a LIMO robot, CSO demonstrated a significant increase in exploration efficiency and completeness compared to state-of-the-art methods. The results highlight CSO's ability to optimize exploration paths and improve overall mapping performance in both simulated and real environments.
strength_weakness: strength_weakness: 1. This paper is well written, and the figures are well plotted. 
2. The method presents a novel architecture of how to introduce constraints based on action mask-based to improve the pipeline based on entropy optimization, which could easily fall into a local minimum and thus cause inefficiency.
3. The results adequately prove the efficiency of the method and reach state-of-the-art results. 
4. Ablation studies prove the importance of each part that the authors proposed.
limitations: limitations: 1. Although the methods are well introduced, some parts of intuitions are not well discussed. 
2. Deep reinforcement learning methods generally require significant training time and extensive data. The paper does not discuss the training duration and the amount of data required, which could be a practical limitation for deployment.
3. I am not an expert in this field, and I will also refer to other reviewers' feedback.
paper_summary: paper_summary: The paper introduces a deep reinforcement learning framework to enhance active scene mapping efficiency. By employing frontiers-based entropy and action mask-based constraints, the method aligns state and critic spaces to avoid long-distance round trips. Evaluations on 29 complex indoor scenes show CSO outperforms existing methods in exploration efficiency and completeness, with additional validation in real-world settings.
strength_weakness: strength_weakness: 1 Introduces a constraint-guided optimization approach using frontiers-based entropy and action mask-based constraints, which is innovative in the field of active scene mapping.
2 Effectively integrates deep reinforcement learning with heuristic rules to optimize state and critic spaces, ensuring better alignment and exploration paths.
3 Extensive experiments conducted in 29 complex indoor scenes using the Matterport3D dataset demonstrate a 19.16% increase in exploration efficiency and a 3.12% improvement in completeness compared to state-of-the-art methods.
limitations: limitations: 1 The integration of heuristic rules such as frontiers-based entropy might limit the flexibility of the approach. Relying on such heuristics could hinder performance in scenarios where these rules are less effective.
2 The paper mentions that the planning time, although better than some traditional methods, is longer than other learning-based methods. This computational overhead might impact real-time applications where rapid decision-making is crucial.
3 While the paper performs ablation studies on action masks and entropy, it lacks a detailed analysis of the contribution of each component of the neural network, such as the Graph Neural Network and PPO components. This would help in understanding the impact of each part of the architecture.
4 The real-world experiments are conducted in a limited number of scenes. Expanding the range and variety of real-world tests would provide a more comprehensive validation of the method’s robustness and generalizability.